[JENKINS] Lucene-Solr-BadApples-7.x-Linux (64bit/jdk-10.0.1) - Build # 49 - Unstable!

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-7.x-Linux (64bit/jdk-10.0.1) - Build # 49 - Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-7.x-Linux/49/
Java: 64bit/jdk-10.0.1 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.cloud.SSLMigrationTest.test

Error Message:
Replica didn't have the proper urlScheme in the ClusterState

Stack Trace:
java.lang.AssertionError: Replica didn't have the proper urlScheme in the ClusterState
        at __randomizedtesting.SeedInfo.seed([A315C3E7C0B1E796:2B41FC3D6E4D8A6E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.SSLMigrationTest.assertReplicaInformation(SSLMigrationTest.java:104)
        at org.apache.solr.cloud.SSLMigrationTest.testMigrateSSL(SSLMigrationTest.java:97)
        at org.apache.solr.cloud.SSLMigrationTest.test(SSLMigrationTest.java:61)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple

Error Message:
IOException occured when talking to server at: http://127.0.0.1:36357/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:36357/solr
        at __randomizedtesting.SeedInfo.seed([A315C3E7C0B1E796:9BA6E719E7423347]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple(AutoAddReplicasPlanActionTest.java:111)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:36357 failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 47 more




Build Log:
[...truncated 12666 lines...]
   [junit4] Suite: org.apache.solr.cloud.SSLMigrationTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/init-core-data-001
   [junit4]   2> 186072 WARN  (SUITE-SSLMigrationTest-seed#[A315C3E7C0B1E796]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2287 numCloses=2287
   [junit4]   2> 186072 INFO  (SUITE-SSLMigrationTest-seed#[A315C3E7C0B1E796]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 186073 INFO  (SUITE-SSLMigrationTest-seed#[A315C3E7C0B1E796]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl="None")
   [junit4]   2> 186073 INFO  (SUITE-SSLMigrationTest-seed#[A315C3E7C0B1E796]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 186073 INFO  (SUITE-SSLMigrationTest-seed#[A315C3E7C0B1E796]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /sk/y
   [junit4]   2> 186074 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 186074 INFO  (Thread-501) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 186074 INFO  (Thread-501) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 186078 ERROR (Thread-501) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 186174 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.ZkTestServer start zk server on port:38201
   [junit4]   2> 186177 INFO  (zkConnectionManagerCallback-408-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186179 INFO  (zkConnectionManagerCallback-410-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186181 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 186182 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 186183 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 186183 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 186184 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 186184 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 186185 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 186185 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 186186 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 186186 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 186186 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 186187 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 186270 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 10.0.1+10
   [junit4]   2> 186271 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 186271 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 186271 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 186272 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4377abb3{/sk/y,null,AVAILABLE}
   [junit4]   2> 186272 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@167c9911{HTTP/1.1,[http/1.1]}{127.0.0.1:41953}
   [junit4]   2> 186272 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.Server Started @186301ms
   [junit4]   2> 186272 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/sk/y, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/tempDir-001/control/data, hostPort=41953, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/control-001/cores}
   [junit4]   2> 186273 ERROR (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 186273 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 186273 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.5.0
   [junit4]   2> 186273 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 186273 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 186273 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-06-12T03:56:59.147376Z
   [junit4]   2> 186275 INFO  (zkConnectionManagerCallback-412-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186276 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 186276 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/control-001/solr.xml
   [junit4]   2> 186278 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 186278 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 186278 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 186324 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38201/solr
   [junit4]   2> 186325 INFO  (zkConnectionManagerCallback-416-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186329 INFO  (zkConnectionManagerCallback-418-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186365 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 186365 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41953_sk%2Fy
   [junit4]   2> 186365 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.c.Overseer Overseer (id=72185843443957764-127.0.0.1:41953_sk%2Fy-n_0000000000) starting
   [junit4]   2> 186373 INFO  (zkConnectionManagerCallback-425-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186375 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38201/solr ready
   [junit4]   2> 186382 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41953_sk%2Fy
   [junit4]   2> 186398 INFO  (zkCallback-417-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 186404 INFO  (zkCallback-424-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 186404 WARN  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
   [junit4]   2> 186418 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 186426 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 186427 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 186428 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/control-001/cores
   [junit4]   2> 186440 INFO  (zkConnectionManagerCallback-431-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 186441 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 186442 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38201/solr ready
   [junit4]   2> 186443 INFO  (qtp1024094747-1964) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:41953_sk%252Fy&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 186444 INFO  (OverseerThreadFactory-698-thread-1) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 186549 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 186549 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 187589 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
   [junit4]   2> 187596 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 187598 WARN  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 187600 WARN  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieDoubleField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 187603 WARN  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieDateField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 187641 WARN  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.EnumField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 187648 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 187660 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 187660 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 187660 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 187661 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 187663 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=24, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 187665 WARN  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 187694 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 187694 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 187695 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 187695 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 187696 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=16, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 187696 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4b8bb247[control_collection_shard1_replica_n1] main]
   [junit4]   2> 187697 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 187697 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 187698 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 187698 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1603037634832105472
   [junit4]   2> 187698 INFO  (searcherExecutor-703-thread-1-processing-n:127.0.0.1:41953_sk%2Fy x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@4b8bb247[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 187701 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 187702 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 187702 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 187702 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41953/sk/y/control_collection_shard1_replica_n1/
   [junit4]   2> 187703 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 187703 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:41953/sk/y/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 187703 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 187704 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41953/sk/y/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 187806 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 187807 INFO  (qtp1024094747-1959) [n:127.0.0.1:41953_sk%2Fy c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1257
   [junit4]   2> 187808 INFO  (qtp1024094747-1964) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 187907 INFO  (zkCallback-417-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 188445 INFO  (OverseerCollectionConfigSetProcessor-72185843443957764-127.0.0.1:41953_sk%2Fy-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 188808 INFO  (qtp1024094747-1964) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:41953_sk%252Fy&wt=javabin&version=2} status=0 QTime=2365
   [junit4]   2> 188812 INFO  (zkConnectionManagerCallback-436-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 188813 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 188813 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38201/solr ready
   [junit4]   2> 188813 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 188814 INFO  (qtp1024094747-1960) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 188816 INFO  (OverseerThreadFactory-698-thread-2) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 188817 WARN  (OverseerThreadFactory-698-thread-2) [    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 189019 INFO  (qtp1024094747-1960) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 189019 INFO  (qtp1024094747-1960) [n:127.0.0.1:41953_sk%2Fy    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=205
   [junit4]   2> 189098 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-1-001 of type NRT
   [junit4]   2> 189099 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 10.0.1+10
   [junit4]   2> 189099 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 189099 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 189100 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 189100 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@350b1163{/sk/y,null,AVAILABLE}
   [junit4]   2> 189100 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@870d2d0{HTTP/1.1,[http/1.1]}{127.0.0.1:40543}
   [junit4]   2> 189100 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.Server Started @189129ms
   [junit4]   2> 189100 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/sk/y, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/tempDir-001/jetty1, hostPort=40543, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-1-001/cores}
   [junit4]   2> 189101 ERROR (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 189101 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 189101 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.5.0
   [junit4]   2> 189101 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 189101 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 189101 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-06-12T03:57:01.975133Z
   [junit4]   2> 189102 INFO  (zkConnectionManagerCallback-438-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 189102 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 189102 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-1-001/solr.xml
   [junit4]   2> 189104 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 189104 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 189105 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 189204 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38201/solr
   [junit4]   2> 189205 INFO  (zkConnectionManagerCallback-442-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 189206 INFO  (zkConnectionManagerCallback-444-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 189210 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 189210 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 189211 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 189211 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40543_sk%2Fy
   [junit4]   2> 189212 INFO  (zkCallback-424-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 189212 INFO  (zkCallback-435-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 189212 INFO  (zkCallback-417-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 189212 INFO  (zkCallback-443-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 189221 INFO  (zkConnectionManagerCallback-451-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 189222 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 189222 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38201/solr ready
   [junit4]   2> 189223 WARN  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
   [junit4]   2> 189231 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 189237 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 189237 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 189238 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-1-001/cores
   [junit4]   2> 189253 INFO  (qtp317076144-2022) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:40543_sk%252Fy&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 189254 INFO  (OverseerCollectionConfigSetProcessor-72185843443957764-127.0.0.1:41953_sk%2Fy-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 189254 INFO  (OverseerThreadFactory-698-thread-3) [ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:40543_sk%2Fy for creating new replica
   [junit4]   2> 189256 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy    x:collection1_shard2_replica_n41] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 190266 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
   [junit4]   2> 190273 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
   [junit4]   2> 190325 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 190349 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
   [junit4]   2> 190349 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 190349 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 190349 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
   [junit4]   2> 190351 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=24, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 190352 WARN  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 190375 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 190376 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 190376 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 190376 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 190377 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=16, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 190377 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@6b451feb[collection1_shard2_replica_n41] main]
   [junit4]   2> 190378 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 190379 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 190379 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 190380 INFO  (searcherExecutor-717-thread-1-processing-n:127.0.0.1:40543_sk%2Fy x:collection1_shard2_replica_n41 c:collection1 s:shard2) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@6b451feb[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 190380 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1603037637644386304
   [junit4]   2> 190384 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node42=0}, version=0}
   [junit4]   2> 190385 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 190385 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 190385 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40543/sk/y/collection1_shard2_replica_n41/
   [junit4]   2> 190385 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 190385 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy http://127.0.0.1:40543/sk/y/collection1_shard2_replica_n41/ has no replicas
   [junit4]   2> 190385 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 190387 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40543/sk/y/collection1_shard2_replica_n41/ shard2
   [junit4]   2> 190537 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 190572 INFO  (qtp317076144-2017) [n:127.0.0.1:40543_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1315
   [junit4]   2> 190574 INFO  (qtp317076144-2022) [n:127.0.0.1:40543_sk%2Fy c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:40543_sk%252Fy&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1320
   [junit4]   2> 190644 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-2-001 of type NRT
   [junit4]   2> 190645 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 10.0.1+10
   [junit4]   2> 190646 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 190646 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 190646 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 190646 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7dcac070{/sk/y,null,AVAILABLE}
   [junit4]   2> 190646 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@487bb68c{HTTP/1.1,[http/1.1]}{127.0.0.1:43795}
   [junit4]   2> 190647 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.Server Started @190676ms
   [junit4]   2> 190647 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/sk/y, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/tempDir-001/jetty2, hostPort=43795, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-2-001/cores, replicaType=NRT}
   [junit4]   2> 190647 ERROR (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 190647 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 190647 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.5.0
   [junit4]   2> 190647 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 190647 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 190647 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-06-12T03:57:03.521601Z
   [junit4]   2> 190648 INFO  (zkConnectionManagerCallback-454-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 190649 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 190649 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-2-001/solr.xml
   [junit4]   2> 190651 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 190651 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 190651 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 190678 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38201/solr
   [junit4]   2> 190679 INFO  (zkConnectionManagerCallback-458-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 190681 INFO  (zkConnectionManagerCallback-460-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 190684 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 190685 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 190687 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 190687 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43795_sk%2Fy
   [junit4]   2> 190687 INFO  (zkCallback-450-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 190687 INFO  (zkCallback-424-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 190687 INFO  (zkCallback-443-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 190687 INFO  (zkCallback-417-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 190687 INFO  (zkCallback-435-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 190688 INFO  (zkCallback-459-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 190696 INFO  (zkConnectionManagerCallback-467-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 190697 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 190697 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38201/solr ready
   [junit4]   2> 190698 WARN  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
   [junit4]   2> 190706 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 190712 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 190712 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 190713 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43795_sk%2Fy    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-2-001/cores
   [junit4]   2> 190732 INFO  (qtp317076144-2016) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:43795_sk%252Fy&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 190734 INFO  (OverseerCollectionConfigSetProcessor-72185843443957764-127.0.0.1:41953_sk%2Fy-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 190734 INFO  (OverseerThreadFactory-698-thread-4) [ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:43795_sk%2Fy for creating new replica
   [junit4]   2> 190737 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy    x:collection1_shard1_replica_n43] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 191781 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
   [junit4]   2> 191788 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema [collection1_shard1_replica_n43] Schema name=test
   [junit4]   2> 191860 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 191869 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n43' using configuration from collection collection1, trusted=true
   [junit4]   2> 191869 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n43' (registry 'solr.core.collection1.shard1.replica_n43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 191870 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 191870 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [[collection1_shard1_replica_n43] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-2-001/cores/collection1_shard1_replica_n43], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-2-001/cores/collection1_shard1_replica_n43/data/]
   [junit4]   2> 191871 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=24, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 191872 WARN  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 191911 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 191911 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 191912 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 191912 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 191913 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=16, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 191913 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.SolrIndexSearcher Opening [Searcher@574ab908[collection1_shard1_replica_n43] main]
   [junit4]   2> 191914 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 191915 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 191915 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 191916 INFO  (searcherExecutor-731-thread-1-processing-n:127.0.0.1:43795_sk%2Fy x:collection1_shard1_replica_n43 c:collection1 s:shard1) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [collection1_shard1_replica_n43] Registered new searcher Searcher@574ab908[collection1_shard1_replica_n43] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 191916 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1603037639254999040
   [junit4]   2> 191919 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node44=0}, version=0}
   [junit4]   2> 191920 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 191920 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 191920 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:43795/sk/y/collection1_shard1_replica_n43/
   [junit4]   2> 191920 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 191920 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy http://127.0.0.1:43795/sk/y/collection1_shard1_replica_n43/ has no replicas
   [junit4]   2> 191920 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 191922 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:43795/sk/y/collection1_shard1_replica_n43/ shard1
   [junit4]   2> 192072 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 192075 INFO  (qtp777459656-2063) [n:127.0.0.1:43795_sk%2Fy c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1338
   [junit4]   2> 192077 INFO  (qtp317076144-2016) [n:127.0.0.1:40543_sk%2Fy c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:43795_sk%252Fy&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1344
   [junit4]   2> 192155 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-3-001 of type NRT
   [junit4]   2> 192155 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 10.0.1+10
   [junit4]   2> 192156 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 192156 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 192156 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 192157 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3ad51371{/sk/y,null,AVAILABLE}
   [junit4]   2> 192157 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1bd6e61d{HTTP/1.1,[http/1.1]}{127.0.0.1:43817}
   [junit4]   2> 192157 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.e.j.s.Server Started @192186ms
   [junit4]   2> 192157 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/sk/y, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/tempDir-001/jetty3, hostPort=43817, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-3-001/cores}
   [junit4]   2> 192157 ERROR (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 192157 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 192157 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.5.0
   [junit4]   2> 192158 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 192158 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 192158 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-06-12T03:57:05.032059Z
   [junit4]   2> 192159 INFO  (zkConnectionManagerCallback-470-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 192160 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 192160 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-3-001/solr.xml
   [junit4]   2> 192161 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 192161 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 192162 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 192223 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38201/solr
   [junit4]   2> 192224 INFO  (zkConnectionManagerCallback-474-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 192226 INFO  (zkConnectionManagerCallback-476-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 192229 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 192230 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 192231 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 192231 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43817_sk%2Fy
   [junit4]   2> 192232 INFO  (zkCallback-424-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 192232 INFO  (zkCallback-466-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 192232 INFO  (zkCallback-417-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 192232 INFO  (zkCallback-443-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 192232 INFO  (zkCallback-459-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 192232 INFO  (zkCallback-435-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 192232 INFO  (zkCallback-450-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 192232 INFO  (zkCallback-475-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 192242 INFO  (zkConnectionManagerCallback-483-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 192243 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 192244 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38201/solr ready
   [junit4]   2> 192244 WARN  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
   [junit4]   2> 192254 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 192260 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 192260 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 192261 INFO  (TEST-SSLMigrationTest.test-seed#[A315C3E7C0B1E796]) [n:127.0.0.1:43817_sk%2Fy    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-3-001/cores
   [junit4]   2> 192284 INFO  (qtp317076144-2020) [n:127.0.0.1:40543_sk%2Fy    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:43817_sk%252Fy&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 192286 INFO  (OverseerCollectionConfigSetProcessor-72185843443957764-127.0.0.1:41953_sk%2Fy-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 192286 INFO  (OverseerThreadFactory-698-thread-5) [ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:43817_sk%2Fy for creating new replica
   [junit4]   2> 192288 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy    x:collection1_shard2_replica_n45] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 193331 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
   [junit4]   2> 193344 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.s.IndexSchema [collection1_shard2_replica_n45] Schema name=test
   [junit4]   2> 193443 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 193463 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n45' using configuration from collection collection1, trusted=true
   [junit4]   2> 193463 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n45' (registry 'solr.core.collection1.shard2.replica_n45') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6581dc0a
   [junit4]   2> 193464 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 193464 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.SolrCore [[collection1_shard2_replica_n45] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-3-001/cores/collection1_shard2_replica_n45], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.SSLMigrationTest_A315C3E7C0B1E796-001/shard-3-001/cores/collection1_shard2_replica_n45/data/]
   [junit4]   2> 193466 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=24, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 193468 WARN  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 193511 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 193511 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 193512 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 193512 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 193513 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=16, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 193514 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.s.SolrIndexSearcher Opening [Searcher@29f79df[collection1_shard2_replica_n45] main]
   [junit4]   2> 193515 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 193515 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 193515 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 193515 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1603037640931672064
   [junit4]   2> 193516 INFO  (searcherExecutor-745-thread-1-processing-n:127.0.0.1:43817_sk%2Fy x:collection1_shard2_replica_n45 c:collection1 s:shard2) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.SolrCore [collection1_shard2_replica_n45] Registered new searcher Searcher@29f79df[collection1_shard2_replica_n45] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 193519 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node42=0, core_node46=0}, version=1}
   [junit4]   2> 193520 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_shard2_replica_n45] o.a.s.c.ZkController Core needs to recover:collection1_shard2_replica_n45
   [junit4]   2> 193520 INFO  (updateExecutor-471-thread-1-processing-n:127.0.0.1:43817_sk%2Fy x:collection1_shard2_replica_n45 c:collection1 s:shard2) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 193522 INFO  (qtp1462924004-2101) [n:127.0.0.1:43817_sk%2Fy c:collection1 s:shard2  x:collection1_s

[...truncated too long message...]

.testSimple3.shard3.replica_n4, service url = null, agent id = null] for registry solr.core.testSimple3.shard3.replica_n4 / com.codahale.metrics.MetricRegistry@60b24b16
   [junit4]   2> 2084036 INFO  (coreCloseExecutor-10737-thread-5) [n:127.0.0.1:36357_solr c:testSimple3 s:shard3 r:core_node6 x:testSimple3_shard3_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testSimple3.shard3.leader, tag=73ac725d
   [junit4]   2> 2084036 INFO  (coreCloseExecutor-10737-thread-1) [n:127.0.0.1:36357_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testSimple1.shard2.leader, tag=2c4e7b82
   [junit4]   2> 2084036 INFO  (coreCloseExecutor-10737-thread-4) [n:127.0.0.1:36357_solr c:testSimple3 s:shard2 r:core_node5 x:testSimple3_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testSimple3.shard2.leader, tag=1826975d
   [junit4]   2> 2084038 INFO  (jetty-closer-6385-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2d36dfb0{/solr,null,UNAVAILABLE}
   [junit4]   2> 2084038 INFO  (jetty-closer-6385-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 2084039 ERROR (SUITE-AutoAddReplicasPlanActionTest-seed#[A315C3E7C0B1E796]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2084039 INFO  (SUITE-AutoAddReplicasPlanActionTest-seed#[A315C3E7C0B1E796]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37325 37325
   [junit4]   2> 2084529 INFO  (Thread-6292) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37325 37325
   [junit4]   2> 2084529 WARN  (Thread-6292) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 10 /solr/collections/testSimple1/terms/shard2
   [junit4]   2> 10 /solr/collections/testSimple1/terms/shard1
   [junit4]   2> 9 /solr/clusterprops.json
   [junit4]   2> 9 /solr/aliases.json
   [junit4]   2> 5 /solr/collections/testSimple2/terms/shard1
   [junit4]   2> 4 /solr/security.json
   [junit4]   2> 4 /solr/configs/conf
   [junit4]   2> 3 /solr/collections/testSimple3/terms/shard3
   [junit4]   2> 3 /solr/collections/testSimple3/terms/shard2
   [junit4]   2> 3 /solr/collections/testSimple3/terms/shard1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 11 /solr/collections/testSimple3/state.json
   [junit4]   2> 10 /solr/collections/testSimple1/state.json
   [junit4]   2> 9 /solr/clusterstate.json
   [junit4]   2> 2 /solr/collections/testSimple2/state.json
   [junit4]   2> 2 /solr/collections/testSimple1/leader_elect/shard1/election/72185967030370309-core_node3-n_0000000000
   [junit4]   2> 2 /solr/autoscaling.json
   [junit4]   2> 2 /solr/collections/testSimple1/leader_elect/shard2/election/72185967030370309-core_node7-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/72185967030370309-127.0.0.1:36661_solr-n_0000000001
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 9 /solr/live_nodes
   [junit4]   2> 9 /solr/collections
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_A315C3E7C0B1E796-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1511, maxMBSortInHeap=5.2136451663352394, sim=RandomSimilarity(queryNorm=true): {}, locale=fr-PF, timezone=America/Cordoba
   [junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=8,threads=1,free=341187616,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestJsonRequest, SchemaApiFailureTest, SolrCloudReportersTest, TestNumericTerms64, ScheduledTriggerIntegrationTest, RecoveryAfterSoftCommitTest, TestHdfsCloudBackupRestore, HDFSCollectionsAPITest, TestAnalyzedSuggestions, SearchHandlerTest, TestClassicSimilarityFactory, ConfigureRecoveryStrategyTest, TestSolrXml, DocValuesTest, TestCodecSupport, HdfsUnloadDistributedZkTest, TestMacroExpander, DistributedQueueTest, TestRealTimeGet, TestBulkSchemaConcurrent, TestNestedDocsSort, RollingRestartTest, NumberUtilsTest, CdcrBidirectionalTest, SimpleCollectionCreateDeleteTest, LukeRequestHandlerTest, BitVectorTest, SolrJmxReporterCloudTest, ClusterStateTest, ClassificationUpdateProcessorIntegrationTest, BasicDistributedZk2Test, TestHashQParserPlugin, DistribCursorPagingTest, TestRangeQuery, SliceStateTest, TestSchemaVersionResource, TestMinMaxOnMultiValuedField, TestStressReorder, TestSolr4Spatial2, TestSchemaManager, TestManagedStopFilterFactory, TimeZoneUtilsTest, TestRequestForwarding, Tagger2Test, TestSolrDeletionPolicy2, EnumFieldTest, TestAtomicUpdateErrorCases, MoveReplicaTest, HdfsAutoAddReplicasIntegrationTest, TestSubQueryTransformerDistrib, LeaderElectionIntegrationTest, SortSpecParsingTest, TestCloudJSONFacetSKG, DistributedSpellCheckComponentTest, BlockJoinFacetSimpleTest, SpellingQueryConverterTest, PKIAuthenticationIntegrationTest, TestDistribIDF, TestReRankQParserPlugin, SmileWriterTest, AssignBackwardCompatibilityTest, NodeMutatorTest, AnalysisErrorHandlingTest, ClusterStateUpdateTest, ActionThrottleTest, SolrMetricsIntegrationTest, TermVectorComponentTest, TestInitParams, TestSearcherReuse, TestDynamicURP, TestShardHandlerFactory, TestLRUStatsCache, TestFileDictionaryLookup, TestDistributedMap, TestDocTermOrdsUninvertLimit, TestUtilizeNode, DistributedFacetPivotSmallAdvancedTest, AssignTest, LIROnShardRestartTest, MissingSegmentRecoveryTest, ShowFileRequestHandlerTest, FileBasedSpellCheckerTest, SolrXmlInZkTest, TestJsonFacetRefinement, TestSchemaNameResource, CopyFieldTest, CoreMergeIndexesAdminHandlerTest, TestUtils, TestXmlQParser, TestTestInjection, TestMaxScoreQueryParser, DistributedFacetExistsSmallTest, TestSuggestSpellingConverter, ConvertedLegacyTest, RegexBytesRefFilterTest, BlockCacheTest, CloudExitableDirectoryReaderTest, BadCopyFieldTest, TestTolerantSearch, ShufflingReplicaListTransformerTest, TestStandardQParsers, TestCoreDiscovery, MinimalSchemaTest, DeleteInactiveReplicaTest, TestQuerySenderListener, TestNumericRangeQuery32, UnloadDistributedZkTest, TestBackupRepositoryFactory, LeaderFailoverAfterPartitionTest, SpellPossibilityIteratorTest, UniqFieldsUpdateProcessorFactoryTest, TestConfigSetsAPI, TestLazyCores, PreAnalyzedFieldTest, AtomicUpdateProcessorFactoryTest, ParsingFieldUpdateProcessorsTest, CollectionsAPIAsyncDistributedZkTest, CurrencyFieldTypeTest, MetricsHistoryHandlerTest, HdfsRecoverLeaseTest, CollectionsAPISolrJTest, TestFaceting, CdcrReplicationHandlerTest, UpdateLogTest, PingRequestHandlerTest, CacheHeaderTest, CoreAdminHandlerTest, SharedFSAutoReplicaFailoverTest, TestCollectionAPIs, TestCloudRecovery, RegexBoostProcessorTest, SuggestComponentTest, SolrCmdDistributorTest, SyncSliceTest, RecoveryZkTest, TestReplicationHandler, LeaderElectionTest, ZkCLITest, TestDistributedSearch, TestReload, StatsComponentTest, TestGroupingSearch, BadIndexSchemaTest, BasicFunctionalityTest, TestIndexSearcher, SoftAutoCommitTest, ConnectionManagerTest, DeleteStatusTest, ForceLeaderTest, LeaderVoteWaitTimeoutTest, OverseerRolesTest, PeerSyncReplicationTest, TestCloudDeleteByQuery, TestCloudInspectUtil, TestClusterProperties, TestCryptoKeys, TestExclusionRuleCollectionAccess, TestLeaderElectionWithEmptyReplica, TestLockTree, TestMiniSolrCloudClusterSSL, TestPullReplica, TestRebalanceLeaders, TestSizeLimitedDistributedMap, TestSkipOverseerOperations, TestStressCloudBlindAtomicUpdates, TestTlogReplica, TestRequestStatusCollectionAPI, AutoAddReplicasPlanActionTest]
   [junit4] Completed [600/816 (2!)] on J2 in 12.69s, 1 test, 1 error <<< FAILURES!

[...truncated 44525 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-7.x-Linux/49/consoleText

[repro] Revision: 6140d8be05d1b7aad565f53dd2ee66b984b9a379

[repro] Repro line:  ant test  -Dtestcase=SSLMigrationTest -Dtests.method=test -Dtests.seed=A315C3E7C0B1E796 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ff-MR -Dtests.timezone=Indian/Cocos -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=AutoAddReplicasPlanActionTest -Dtests.method=testSimple -Dtests.seed=A315C3E7C0B1E796 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-PF -Dtests.timezone=America/Cordoba -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       SSLMigrationTest
[repro]       AutoAddReplicasPlanActionTest
[repro] ant compile-test

[...truncated 3323 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 -Dtests.class="*.SSLMigrationTest|*.AutoAddReplicasPlanActionTest" -Dtests.showOutput=onerror  -Dtests.seed=A315C3E7C0B1E796 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ff-MR -Dtests.timezone=Indian/Cocos -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[...truncated 97 lines...]
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.SSLMigrationTest
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
[repro] Exiting with code 0

[...truncated 40 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-7.x-Linux (64bit/jdk-10.0.1) - Build # 50 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-7.x-Linux/50/
Java: 64bit/jdk-10.0.1 -XX:+UseCompressedOops -XX:+UseSerialGC

6 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger

Error Message:
number of ops expected:<2> but was:<1>

Stack Trace:
java.lang.AssertionError: number of ops expected:<2> but was:<1>
        at __randomizedtesting.SeedInfo.seed([E80481B2FD54029C:8BCFB730649B71B1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger(IndexSizeTriggerTest.java:187)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: DocCollection(testSplitIntegration_collection//clusterstate.json/133)={   "replicationFactor":"2",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testSplitIntegration_collection_shard2_replica_n3",           "leader":"true",           "SEARCHER.searcher.maxDoc":11,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":15740,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.4659017324447632E-5,           "SEARCHER.searcher.numDocs":11},         "core_node4":{           "core":"testSplitIntegration_collection_shard2_replica_n4",           "SEARCHER.searcher.maxDoc":11,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":15740,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.4659017324447632E-5,           "SEARCHER.searcher.numDocs":11}},       "range":"0-7fffffff",       "state":"active"},     "shard1":{       "stateTimestamp":"1528958556866825300",       "replicas":{         "core_node1":{           "core":"testSplitIntegration_collection_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14},         "core_node2":{           "core":"testSplitIntegration_collection_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14}},       "range":"80000000-ffffffff",       "state":"inactive"},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1528958556867176600",       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node10":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_1_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10000_solr",           "base_url":"http://127.0.0.1:10000/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7},         "core_node9":{           "core":"testSplitIntegration_collection_shard1_1_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10001_solr",           "base_url":"http://127.0.0.1:10001/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1528958556867112250",       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node7":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_0_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10001_solr",           "base_url":"http://127.0.0.1:10001/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7},         "core_node8":{           "core":"testSplitIntegration_collection_shard1_0_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10000_solr",           "base_url":"http://127.0.0.1:10000/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7}}}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: DocCollection(testSplitIntegration_collection//clusterstate.json/133)={
  "replicationFactor":"2",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testSplitIntegration_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":11,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":15740,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.4659017324447632E-5,
          "SEARCHER.searcher.numDocs":11},
        "core_node4":{
          "core":"testSplitIntegration_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":11,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":15740,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.4659017324447632E-5,
          "SEARCHER.searcher.numDocs":11}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1528958556866825300",
      "replicas":{
        "core_node1":{
          "core":"testSplitIntegration_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14},
        "core_node2":{
          "core":"testSplitIntegration_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1528958556867176600",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10000_solr",
          "base_url":"http://127.0.0.1:10000/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node9":{
          "core":"testSplitIntegration_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10001_solr",
          "base_url":"http://127.0.0.1:10001/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1528958556867112250",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10001_solr",
          "base_url":"http://127.0.0.1:10001/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node8":{
          "core":"testSplitIntegration_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10000_solr",
          "base_url":"http://127.0.0.1:10000/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7}}}}}
        at __randomizedtesting.SeedInfo.seed([E80481B2FD54029C:D18A38F2D2ABCB62]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration(IndexSizeTriggerTest.java:299)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds

Error Message:
failed to create testMixedBounds_collection Live Nodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr] Last available state: DocCollection(testMixedBounds_collection//clusterstate.json/25)={   "replicationFactor":"2",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testMixedBounds_collection_shard2_replica_n3",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node4":{           "core":"testMixedBounds_collection_shard2_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"0-7fffffff",       "state":"active"},     "shard1":{       "stateTimestamp":"1528958482236382850",       "replicas":{         "core_node1":{           "core":"testMixedBounds_collection_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":495,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":257740,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.4003908038139343E-4,           "SEARCHER.searcher.numDocs":495},         "core_node2":{           "core":"testMixedBounds_collection_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":495,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":257740,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.4003908038139343E-4,           "SEARCHER.searcher.numDocs":495}},       "range":"80000000-ffffffff",       "state":"inactive"},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1528958482251234700",       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node10":{           "leader":"true",           "core":"testMixedBounds_collection_shard1_1_replica1",           "SEARCHER.searcher.maxDoc":247,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":133740,           "node_name":"127.0.0.1:10000_solr",           "base_url":"http://127.0.0.1:10000/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.245550811290741E-4,           "SEARCHER.searcher.numDocs":247},         "core_node9":{           "core":"testMixedBounds_collection_shard1_1_replica0",           "SEARCHER.searcher.maxDoc":247,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":133740,           "node_name":"127.0.0.1:10001_solr",           "base_url":"http://127.0.0.1:10001/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.245550811290741E-4,           "SEARCHER.searcher.numDocs":247}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1528958482251013800",       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node7":{           "leader":"true",           "core":"testMixedBounds_collection_shard1_0_replica0",           "SEARCHER.searcher.maxDoc":248,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":144480,           "node_name":"127.0.0.1:10001_solr",           "base_url":"http://127.0.0.1:10001/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.3455748558044434E-4,           "SEARCHER.searcher.numDocs":248},         "core_node8":{           "core":"testMixedBounds_collection_shard1_0_replica1",           "SEARCHER.searcher.maxDoc":248,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":144480,           "node_name":"127.0.0.1:10000_solr",           "base_url":"http://127.0.0.1:10000/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.3455748558044434E-4,           "SEARCHER.searcher.numDocs":248}}}}}

Stack Trace:
java.lang.AssertionError: failed to create testMixedBounds_collection
Live Nodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
Last available state: DocCollection(testMixedBounds_collection//clusterstate.json/25)={
  "replicationFactor":"2",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testMixedBounds_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node4":{
          "core":"testMixedBounds_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1528958482236382850",
      "replicas":{
        "core_node1":{
          "core":"testMixedBounds_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495},
        "core_node2":{
          "core":"testMixedBounds_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1528958482251234700",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10000_solr",
          "base_url":"http://127.0.0.1:10000/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247},
        "core_node9":{
          "core":"testMixedBounds_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10001_solr",
          "base_url":"http://127.0.0.1:10001/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1528958482251013800",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10001_solr",
          "base_url":"http://127.0.0.1:10001/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248},
        "core_node8":{
          "core":"testMixedBounds_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10000_solr",
          "base_url":"http://127.0.0.1:10000/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248}}}}}
        at __randomizedtesting.SeedInfo.seed([E80481B2FD54029C:E2873E1FB0EF09C6]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:69)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds(IndexSizeTriggerTest.java:559)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.util.concurrent.TimeoutException: last state: DocCollection(testMixedBounds_collection//clusterstate.json/25)={
  "replicationFactor":"2",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testMixedBounds_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node4":{
          "core":"testMixedBounds_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1528958482236382850",
      "replicas":{
        "core_node1":{
          "core":"testMixedBounds_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495},
        "core_node2":{
          "core":"testMixedBounds_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1528958482251234700",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10000_solr",
          "base_url":"http://127.0.0.1:10000/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247},
        "core_node9":{
          "core":"testMixedBounds_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10001_solr",
          "base_url":"http://127.0.0.1:10001/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1528958482251013800",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10001_solr",
          "base_url":"http://127.0.0.1:10001/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248},
        "core_node8":{
          "core":"testMixedBounds_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10000_solr",
          "base_url":"http://127.0.0.1:10000/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248}}}}}
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:63)
        ... 39 more


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: DocCollection(testSplitIntegration_collection//clusterstate.json/134)={   "replicationFactor":"2",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testSplitIntegration_collection_shard2_replica_n3",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node4":{           "core":"testSplitIntegration_collection_shard2_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"0-7fffffff",       "state":"active"},     "shard1":{       "stateTimestamp":"1528958648794876250",       "replicas":{         "core_node1":{           "core":"testSplitIntegration_collection_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14},         "core_node2":{           "core":"testSplitIntegration_collection_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14}},       "range":"80000000-ffffffff",       "state":"inactive"},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1528958648795225450",       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node10":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_1_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10000_solr",           "base_url":"http://127.0.0.1:10000/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7},         "core_node9":{           "core":"testSplitIntegration_collection_shard1_1_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10001_solr",           "base_url":"http://127.0.0.1:10001/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1528958648795165200",       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node7":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_0_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10001_solr",           "base_url":"http://127.0.0.1:10001/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7},         "core_node8":{           "core":"testSplitIntegration_collection_shard1_0_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10000_solr",           "base_url":"http://127.0.0.1:10000/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7}}}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: DocCollection(testSplitIntegration_collection//clusterstate.json/134)={
  "replicationFactor":"2",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testSplitIntegration_collection_shard2_replica_n3",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node4":{
          "core":"testSplitIntegration_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1528958648794876250",
      "replicas":{
        "core_node1":{
          "core":"testSplitIntegration_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14},
        "core_node2":{
          "core":"testSplitIntegration_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1528958648795225450",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10000_solr",
          "base_url":"http://127.0.0.1:10000/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node9":{
          "core":"testSplitIntegration_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10001_solr",
          "base_url":"http://127.0.0.1:10001/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1528958648795165200",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10001_solr",
          "base_url":"http://127.0.0.1:10001/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node8":{
          "core":"testSplitIntegration_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10000_solr",
          "base_url":"http://127.0.0.1:10000/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7}}}}}
        at __randomizedtesting.SeedInfo.seed([E80481B2FD54029C:D18A38F2D2ABCB62]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration(IndexSizeTriggerTest.java:299)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: DocCollection(testSplitIntegration_collection//clusterstate.json/128)={   "replicationFactor":"2",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testSplitIntegration_collection_shard2_replica_n3",           "leader":"true",           "SEARCHER.searcher.maxDoc":11,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":15740,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.4659017324447632E-5,           "SEARCHER.searcher.numDocs":11},         "core_node4":{           "core":"testSplitIntegration_collection_shard2_replica_n4",           "SEARCHER.searcher.maxDoc":11,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":15740,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.4659017324447632E-5,           "SEARCHER.searcher.numDocs":11}},       "range":"0-7fffffff",       "state":"active"},     "shard1":{       "stateTimestamp":"1528958752484693450",       "replicas":{         "core_node1":{           "core":"testSplitIntegration_collection_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14},         "core_node2":{           "core":"testSplitIntegration_collection_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14}},       "range":"80000000-ffffffff",       "state":"inactive"},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1528958752485703100",       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node10":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_1_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10002_solr",           "base_url":"http://127.0.0.1:10002/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7},         "core_node9":{           "core":"testSplitIntegration_collection_shard1_1_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10003_solr",           "base_url":"http://127.0.0.1:10003/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1528958752485634350",       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node7":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_0_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10003_solr",           "base_url":"http://127.0.0.1:10003/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7},         "core_node8":{           "core":"testSplitIntegration_collection_shard1_0_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10002_solr",           "base_url":"http://127.0.0.1:10002/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7}}}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: DocCollection(testSplitIntegration_collection//clusterstate.json/128)={
  "replicationFactor":"2",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testSplitIntegration_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":11,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":15740,
          "node_name":"127.0.0.1:10003_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.4659017324447632E-5,
          "SEARCHER.searcher.numDocs":11},
        "core_node4":{
          "core":"testSplitIntegration_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":11,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":15740,
          "node_name":"127.0.0.1:10002_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.4659017324447632E-5,
          "SEARCHER.searcher.numDocs":11}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1528958752484693450",
      "replicas":{
        "core_node1":{
          "core":"testSplitIntegration_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10003_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14},
        "core_node2":{
          "core":"testSplitIntegration_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10002_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1528958752485703100",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10002_solr",
          "base_url":"http://127.0.0.1:10002/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node9":{
          "core":"testSplitIntegration_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10003_solr",
          "base_url":"http://127.0.0.1:10003/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1528958752485634350",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10003_solr",
          "base_url":"http://127.0.0.1:10003/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node8":{
          "core":"testSplitIntegration_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10002_solr",
          "base_url":"http://127.0.0.1:10002/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7}}}}}
        at __randomizedtesting.SeedInfo.seed([E80481B2FD54029C:D18A38F2D2ABCB62]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration(IndexSizeTriggerTest.java:299)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: DocCollection(testSplitIntegration_collection//clusterstate.json/132)={   "replicationFactor":"2",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testSplitIntegration_collection_shard2_replica_n3",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10129_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node4":{           "core":"testSplitIntegration_collection_shard2_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10128_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"0-7fffffff",       "state":"active"},     "shard1":{       "stateTimestamp":"1528976078786538600",       "replicas":{         "core_node1":{           "core":"testSplitIntegration_collection_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10129_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14},         "core_node2":{           "core":"testSplitIntegration_collection_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10128_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14}},       "range":"80000000-ffffffff",       "state":"inactive"},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1528976078787305350",       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node10":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_1_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10128_solr",           "base_url":"http://127.0.0.1:10128/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7},         "core_node9":{           "core":"testSplitIntegration_collection_shard1_1_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10129_solr",           "base_url":"http://127.0.0.1:10129/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1528976078787178950",       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node7":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_0_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10129_solr",           "base_url":"http://127.0.0.1:10129/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7},         "core_node8":{           "core":"testSplitIntegration_collection_shard1_0_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10128_solr",           "base_url":"http://127.0.0.1:10128/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7}}}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: DocCollection(testSplitIntegration_collection//clusterstate.json/132)={
  "replicationFactor":"2",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testSplitIntegration_collection_shard2_replica_n3",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10129_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node4":{
          "core":"testSplitIntegration_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10128_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1528976078786538600",
      "replicas":{
        "core_node1":{
          "core":"testSplitIntegration_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10129_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14},
        "core_node2":{
          "core":"testSplitIntegration_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10128_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1528976078787305350",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10128_solr",
          "base_url":"http://127.0.0.1:10128/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node9":{
          "core":"testSplitIntegration_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10129_solr",
          "base_url":"http://127.0.0.1:10129/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1528976078787178950",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10129_solr",
          "base_url":"http://127.0.0.1:10129/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node8":{
          "core":"testSplitIntegration_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10128_solr",
          "base_url":"http://127.0.0.1:10128/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7}}}}}
        at __randomizedtesting.SeedInfo.seed([E80481B2FD54029C:D18A38F2D2ABCB62]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration(IndexSizeTriggerTest.java:299)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 12884 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.IndexSizeTriggerTest_E80481B2FD54029C-001/init-core-data-001
   [junit4]   2> 664309 INFO  (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 664309 INFO  (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 664310 INFO  (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 664310 INFO  (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.IndexSizeTriggerTest_E80481B2FD54029C-001/tempDir-001
   [junit4]   2> 664310 INFO  (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 664320 INFO  (Thread-3626) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 664320 INFO  (Thread-3626) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 664356 ERROR (Thread-3626) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 664420 INFO  (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:42259
   [junit4]   2> 664444 INFO  (zkConnectionManagerCallback-1844-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664452 INFO  (jetty-launcher-1841-thread-1) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 10.0.1+10
   [junit4]   2> 664453 INFO  (jetty-launcher-1841-thread-2) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 10.0.1+10
   [junit4]   2> 664471 INFO  (jetty-launcher-1841-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 664471 INFO  (jetty-launcher-1841-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 664471 INFO  (jetty-launcher-1841-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 664485 INFO  (jetty-launcher-1841-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@62968a2b{/solr,null,AVAILABLE}
   [junit4]   2> 664486 INFO  (jetty-launcher-1841-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2ca57f0a{SSL,[ssl, http/1.1]}{127.0.0.1:۴۳۰۶۳}
   [junit4]   2> 664487 INFO  (jetty-launcher-1841-thread-1) [    ] o.e.j.s.Server Started @۶۶۴۵۰۸ms
   [junit4]   2> 664487 INFO  (jetty-launcher-1841-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43063}
   [junit4]   2> 664487 ERROR (jetty-launcher-1841-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 664487 INFO  (jetty-launcher-1841-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 664487 INFO  (jetty-launcher-1841-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.5.0
   [junit4]   2> 664487 INFO  (jetty-launcher-1841-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 664487 INFO  (jetty-launcher-1841-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 664487 INFO  (jetty-launcher-1841-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-06-14T06:00:18.094544Z
   [junit4]   2> 664504 INFO  (jetty-launcher-1841-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 664504 INFO  (jetty-launcher-1841-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 664504 INFO  (jetty-launcher-1841-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 664505 INFO  (jetty-launcher-1841-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@651413b5{/solr,null,AVAILABLE}
   [junit4]   2> 664505 INFO  (jetty-launcher-1841-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2f59bab{SSL,[ssl, http/1.1]}{127.0.0.1:۴۱۹۸۹}
   [junit4]   2> 664505 INFO  (jetty-launcher-1841-thread-2) [    ] o.e.j.s.Server Started @۶۶۴۵۲۷ms
   [junit4]   2> 664505 INFO  (jetty-launcher-1841-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41989}
   [junit4]   2> 664505 ERROR (jetty-launcher-1841-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 664505 INFO  (jetty-launcher-1841-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 664505 INFO  (jetty-launcher-1841-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.5.0
   [junit4]   2> 664506 INFO  (jetty-launcher-1841-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 664506 INFO  (jetty-launcher-1841-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 664506 INFO  (jetty-launcher-1841-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-06-14T06:00:18.113070Z
   [junit4]   2> 664507 INFO  (zkConnectionManagerCallback-1846-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664507 INFO  (jetty-launcher-1841-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 664533 INFO  (zkConnectionManagerCallback-1848-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664533 INFO  (jetty-launcher-1841-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 664595 INFO  (jetty-launcher-1841-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42259/solr
   [junit4]   2> 664624 INFO  (zkConnectionManagerCallback-1852-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664632 INFO  (zkConnectionManagerCallback-1854-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664657 INFO  (jetty-launcher-1841-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42259/solr
   [junit4]   2> 664676 INFO  (zkConnectionManagerCallback-1860-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664697 INFO  (zkConnectionManagerCallback-1862-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664702 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 664702 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41989_solr
   [junit4]   2> 664703 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.c.Overseer Overseer (id=72197652964704262-127.0.0.1:41989_solr-n_0000000000) starting
   [junit4]   2> 664718 INFO  (jetty-launcher-1841-thread-1) [n:127.0.0.1:43063_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 664719 INFO  (jetty-launcher-1841-thread-1) [n:127.0.0.1:43063_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43063_solr
   [junit4]   2> 664728 INFO  (zkCallback-1861-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 664735 INFO  (zkCallback-1853-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 664746 INFO  (zkConnectionManagerCallback-1869-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664747 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 664747 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42259/solr ready
   [junit4]   2> 664761 DEBUG (OverseerAutoScalingTriggerThread-72197652964704262-127.0.0.1:41989_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 664762 DEBUG (OverseerAutoScalingTriggerThread-72197652964704262-127.0.0.1:41989_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 664763 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 664763 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41989_solr
   [junit4]   2> 664764 DEBUG (OverseerAutoScalingTriggerThread-72197652964704262-127.0.0.1:41989_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 664765 DEBUG (OverseerAutoScalingTriggerThread-72197652964704262-127.0.0.1:41989_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 664768 INFO  (zkCallback-1853-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 664777 INFO  (zkConnectionManagerCallback-1874-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664777 INFO  (zkCallback-1861-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 664777 INFO  (jetty-launcher-1841-thread-1) [n:127.0.0.1:43063_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 664778 INFO  (jetty-launcher-1841-thread-1) [n:127.0.0.1:43063_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42259/solr ready
   [junit4]   2> 664782 DEBUG (OverseerAutoScalingTriggerThread-72197652964704262-127.0.0.1:41989_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:43063_solr]
   [junit4]   2> 664796 DEBUG (OverseerAutoScalingTriggerThread-72197652964704262-127.0.0.1:41989_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 664797 INFO  (zkCallback-1868-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 664797 DEBUG (OverseerAutoScalingTriggerThread-72197652964704262-127.0.0.1:41989_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 664797 DEBUG (ScheduledTrigger-2693-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 664807 INFO  (jetty-launcher-1841-thread-1) [n:127.0.0.1:43063_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 664827 INFO  (jetty-launcher-1841-thread-1) [n:127.0.0.1:43063_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43063.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3db2651b
   [junit4]   2> 664835 INFO  (jetty-launcher-1841-thread-1) [n:127.0.0.1:43063_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43063.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3db2651b
   [junit4]   2> 664836 INFO  (jetty-launcher-1841-thread-1) [n:127.0.0.1:43063_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43063.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3db2651b
   [junit4]   2> 664836 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 664837 INFO  (jetty-launcher-1841-thread-1) [n:127.0.0.1:43063_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.IndexSizeTriggerTest_E80481B2FD54029C-001/tempDir-001/node1/.
   [junit4]   2> 664849 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41989.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3db2651b
   [junit4]   2> 664869 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41989.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3db2651b
   [junit4]   2> 664869 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41989.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3db2651b
   [junit4]   2> 664870 INFO  (jetty-launcher-1841-thread-2) [n:127.0.0.1:41989_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.IndexSizeTriggerTest_E80481B2FD54029C-001/tempDir-001/node2/.
   [junit4]   2> 664897 INFO  (zkConnectionManagerCallback-1878-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664908 INFO  (zkConnectionManagerCallback-1883-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 664909 INFO  (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 664909 INFO  (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42259/solr ready
   [junit4]   2> 664936 DEBUG (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10128_solr
   [junit4]   2> 664937 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 664937 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 664937 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Curre

[...truncated too long message...]


   [junit4]   2> 13172 INFO  (jetty-closer-45-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@14689f33: rootName = solr_35319, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@43cd5137
   [junit4]   2> 13172 INFO  (jetty-closer-45-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2d1bc303: rootName = solr_43359, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@2c96762b
   [junit4]   2> 13177 INFO  (jetty-closer-45-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 13177 INFO  (jetty-closer-45-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4e3ceb37: rootName = solr_35319, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@582889e2
   [junit4]   2> 13182 INFO  (jetty-closer-45-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 13182 INFO  (jetty-closer-45-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@77c9b38f: rootName = solr_35319, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@d83dc3c
   [junit4]   2> 13183 INFO  (jetty-closer-45-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 13183 INFO  (jetty-closer-45-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@50c3e81a: rootName = solr_43359, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@582889e2
   [junit4]   2> 13183 INFO  (jetty-closer-45-thread-2) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:35319_solr
   [junit4]   2> 13185 INFO  (jetty-closer-45-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 13185 INFO  (zkCallback-64-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 13185 INFO  (zkCallback-56-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 13185 INFO  (zkCallback-71-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 13188 INFO  (zkCallback-77-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 13188 INFO  (jetty-closer-45-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 13189 INFO  (jetty-closer-45-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7a6f2ae: rootName = solr_43359, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@d83dc3c
   [junit4]   2> 13189 INFO  (jetty-closer-45-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6ac344b1{/solr,null,UNAVAILABLE}
   [junit4]   2> 13189 INFO  (jetty-closer-45-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 13190 INFO  (jetty-closer-45-thread-1) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:43359_solr
   [junit4]   2> 13191 INFO  (jetty-closer-45-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 13191 INFO  (jetty-closer-45-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 13191 INFO  (zkCallback-71-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 13191 INFO  (zkCallback-64-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 13192 INFO  (jetty-closer-45-thread-1) [    ] o.a.s.c.Overseer Overseer (id=72197811028819974-127.0.0.1:43359_solr-n_0000000000) closing
   [junit4]   2> 13192 INFO  (OverseerStateUpdate-72197811028819974-127.0.0.1:43359_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:43359_solr
   [junit4]   2> 13192 WARN  (OverseerAutoScalingTriggerThread-72197811028819974-127.0.0.1:43359_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 13195 DEBUG (jetty-closer-45-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 13195 DEBUG (jetty-closer-45-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 13195 DEBUG (jetty-closer-45-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 13195 DEBUG (jetty-closer-45-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 13195 DEBUG (jetty-closer-45-thread-1) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 13195 DEBUG (jetty-closer-45-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 13196 INFO  (jetty-closer-45-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@dd0e5dc{/solr,null,UNAVAILABLE}
   [junit4]   2> 13196 INFO  (jetty-closer-45-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 13197 ERROR (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 13197 INFO  (SUITE-IndexSizeTriggerTest-seed#[E80481B2FD54029C]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36045 36045
   [junit4]   2> 13321 INFO  (Thread-20) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36045 36045
   [junit4]   2> 13321 WARN  (Thread-20) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/aliases.json
   [junit4]   2> 2 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/clusterstate.json
   [junit4]   2> 5 /solr/clusterprops.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/live_nodes
   [junit4]   2> 5 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.IndexSizeTriggerTest_E80481B2FD54029C-002
   [junit4]   2> Jun 14, 2018 6:40:33 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1680, maxMBSortInHeap=6.393579083888666, sim=RandomSimilarity(queryNorm=true): {}, locale=fa-IR, timezone=Australia/LHI
   [junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=8,threads=1,free=424403200,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [IndexSizeTriggerTest, IndexSizeTriggerTest]
   [junit4] Completed [4/5 (3!)] on J1 in 3.81s, 4 tests, 1 error <<< FAILURES!

[...truncated 22 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/common-build.xml:1568: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/common-build.xml:1092: There were test failures: 5 suites, 20 tests, 3 errors, 2 failures [seed: E80481B2FD54029C]

Total time: 17 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   3/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=35a7e95bce54d53820ce3e0b1e2966f609a1c1d2, workspace=/var/lib/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #49
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-7.x-Linux (64bit/jdk-10.0.1) - Build # 51 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-7.x-Linux/51/
Java: 64bit/jdk-10.0.1 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
did not finish processing all events in time: started=5, finished=4

Stack Trace:
java.lang.AssertionError: did not finish processing all events in time: started=5, finished=4
        at __randomizedtesting.SeedInfo.seed([1961A9D5677685CF:A674672BE49CE049]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.doTestNodeLost(TestLargeCluster.java:522)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost(TestLargeCluster.java:375)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
did not finish processing all events in time: started=5, finished=4

Stack Trace:
java.lang.AssertionError: did not finish processing all events in time: started=5, finished=4
        at __randomizedtesting.SeedInfo.seed([1961A9D5677685CF:A674672BE49CE049]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.doTestNodeLost(TestLargeCluster.java:522)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost(TestLargeCluster.java:375)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 13424 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestLargeCluster_1961A9D5677685CF-001/init-core-data-001
   [junit4]   2> 1025244 INFO  (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1025245 INFO  (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 1025245 INFO  (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1025255 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 1025255 DEBUG (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10000_solr
   [junit4]   2> 1025255 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 1025255 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 1025255 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 1025256 INFO  (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1025257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10017_solr, 127.0.0.1:10002_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr, 127.0.0.1:10015_solr, 127.0.0.1:10006_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr, 127.0.0.1:10013_solr, 127.0.0.1:10000_solr, 127.0.0.1:10008_solr, 127.0.0.1:10014_solr, 127.0.0.1:10003_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 1025257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1025257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 1025257 DEBUG (ScheduledTrigger-6289-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 28
   [junit4]   2> 1025262 INFO  (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testBasic
   [junit4]   2> 1025262 DEBUG (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1025263 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1025263 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1025263 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1025263 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1025263 INFO  (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 1025263 DEBUG (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 1025264 DEBUG (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 1025264 DEBUG (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 1025264 DEBUG (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 1025264 DEBUG (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 1025264 DEBUG (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1025263 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2050) ~[?:?]
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2084) ~[?:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:844) [?:?]
   [junit4]   2> 1025264 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 1025264 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 1025264 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 1025264 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 1025265 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10001_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10000_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10002_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10037_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr]
   [junit4]   2> 1025265 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1025265 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1025265 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 1025285 DEBUG (ScheduledTrigger-6294-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1025305 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1025326 DEBUG (ScheduledTrigger-6294-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1025346 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1025366 DEBUG (ScheduledTrigger-6294-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1025371 DEBUG (simCloudManagerPool-6293-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1025371 DEBUG (simCloudManagerPool-6293-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 1025371 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1025371 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1025371 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1025375 DEBUG (simCloudManagerPool-6293-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1025375 DEBUG (simCloudManagerPool-6293-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 1025375 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 1025375 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1025375 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1025375 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 1025403 DEBUG (simCloudManagerPool-6293-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for .system / shard1: {"core_node1":{
   [junit4]   2>     "core":".system_shard1_replica_n1",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "node_name":"127.0.0.1:10000_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":".system"}}
   [junit4]   2> 1025406 DEBUG (simCloudManagerPool-6293-thread-5) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1025406 DEBUG (simCloudManagerPool-6293-thread-5) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 1025407 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 1025407 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger1 - Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10001_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10000_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10002_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10037_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr]
   [junit4]   2> 1025407 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1025407 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 1025407 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025409 DEBUG (simCloudManagerPool-6293-thread-6) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1025409 DEBUG (simCloudManagerPool-6293-thread-6) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 1025409 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 1025409 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1025409 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 1025427 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025448 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025468 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025488 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025508 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025529 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025549 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025569 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025589 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025610 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025630 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025650 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025670 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025691 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025711 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025731 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025751 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025772 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025792 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025812 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025832 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025853 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025873 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025893 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025913 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025934 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025954 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025974 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1025994 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026015 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026035 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026055 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026075 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026095 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026116 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026136 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026156 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026176 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026196 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026217 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026237 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026257 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026277 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 100
   [junit4]   2> 1026280 DEBUG (simCloudManagerPool-6293-thread-85) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testBasic / shard4: {"core_node54":{
   [junit4]   2>     "core":"testBasic_shard4_replica_t54",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "node_name":"127.0.0.1:10055_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"TLOG",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard4",
   [junit4]   2>     "collection":"testBasic"}}
   [junit4]   2> 1026280 DEBUG (simCloudManagerPool-6293-thread-84) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testBasic / shard3: {"core_node39":{
   [junit4]   2>     "core":"testBasic_shard3_replica_t39",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "node_name":"127.0.0.1:10049_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"TLOG",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard3",
   [junit4]   2>     "collection":"testBasic"}}
   [junit4]   2> 1026280 DEBUG (simCloudManagerPool-6293-thread-83) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testBasic / shard2: {"core_node18":{
   [junit4]   2>     "core":"testBasic_shard2_replica_n18",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "node_name":"127.0.0.1:10052_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard2",
   [junit4]   2>     "collection":"testBasic"}}
   [junit4]   2> 1026280 DEBUG (simCloudManagerPool-6293-thread-86) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testBasic / shard5: {"core_node65":{
   [junit4]   2>     "core":"testBasic_shard5_replica_n65",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "node_name":"127.0.0.1:10087_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard5",
   [junit4]   2>     "collection":"testBasic"}}
   [junit4]   2> 1026280 DEBUG (simCloudManagerPool-6293-thread-87) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testBasic / shard1: {"core_node1":{
   [junit4]   2>     "core":"testBasic_shard1_replica_n1",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "node_name":"127.0.0.1:10054_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"testBasic"}}
   [junit4]   2> 1026291 INFO  (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.TestLargeCluster Ready after 618ms
   [junit4]   2> 1026297 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 97
   [junit4]   2> 1026298 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10082_solr
   [junit4]   2> 1026298 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10077_solr
   [junit4]   2> 1026298 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10026_solr
   [junit4]   2> 1026318 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 94
   [junit4]   2> 1026318 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10016_solr
   [junit4]   2> 1026318 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10083_solr
   [junit4]   2> 1026318 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10058_solr
   [junit4]   2> 1026327 INFO  (simCloudManagerPool-6293-thread-119) [    ] o.a.s.c.ActionThrottle Throttling leader attempts - waiting for 2581ms
   [junit4]   2> 1026338 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 92
   [junit4]   2> 1026338 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10049_solr
   [junit4]   2> 1026338 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10073_solr
   [junit4]   2> 1026358 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 92
   [junit4]   2> 1026379 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 92
   [junit4]   2> 1026399 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 92
   [junit4]   2> 1026399 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10082_solr, 127.0.0.1:10077_solr, 127.0.0.1:10026_solr]
   [junit4]   2> 1026399 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"82ed6c9f3f457T9pw511u6t00f16ou7eh7prv38",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303299859772503,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303299859772503,
   [junit4]   2>       2303299860425653,
   [junit4]   2>       2303299860874453],
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10082_solr",
   [junit4]   2>       "127.0.0.1:10077_solr",
   [junit4]   2>       "127.0.0.1:10026_solr"]}}
   [junit4]   2> 1026399 DEBUG (ScheduledTrigger-6294-thread-4) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_trigger1]
   [junit4]   2> 1026402 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"82ed6c9f3f457T9pw511u6t00f16ou7eh7prv38",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303299859772503,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303299859772503,
   [junit4]   2>       2303299860425653,
   [junit4]   2>       2303299860874453],
   [junit4]   2>     "_enqueue_time_":2303304955681953,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10082_solr",
   [junit4]   2>       "127.0.0.1:10077_solr",
   [junit4]   2>       "127.0.0.1:10026_solr"]}}
   [junit4]   2> 1026403 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction -- processing event: {
   [junit4]   2>   "id":"82ed6c9f3f457T9pw511u6t00f16ou7eh7prv38",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303299859772503,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303299859772503,
   [junit4]   2>       2303299860425653,
   [junit4]   2>       2303299860874453],
   [junit4]   2>     "_enqueue_time_":2303304955681953,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10082_solr",
   [junit4]   2>       "127.0.0.1:10077_solr",
   [junit4]   2>       "127.0.0.1:10026_solr"]}} with context properties: {AFTER_ACTION=[start], BEFORE_ACTION=[start, compute]}
   [junit4]   2> 1026450 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10072_solr&inPlaceMove=true&replica=core_node74
   [junit4]   2> 1026476 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10056_solr&inPlaceMove=true&replica=core_node29
   [junit4]   2> 1026488 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10059_solr&inPlaceMove=true&replica=core_node32
   [junit4]   2> 1026488 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2303309395 sessionWrapper.createTime 2303305196939953, this.sessionWrapper.createTime 2303305196939953
   [junit4]   2> 1026488 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction -- processing event: {
   [junit4]   2>   "id":"82ed6c9f3f457T9pw511u6t00f16ou7eh7prv38",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303299859772503,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303299859772503,
   [junit4]   2>       2303299860425653,
   [junit4]   2>       2303299860874453],
   [junit4]   2>     "_enqueue_time_":2303304955681953,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10082_solr",
   [junit4]   2>       "127.0.0.1:10077_solr",
   [junit4]   2>       "127.0.0.1:10026_solr"]}} with context properties: {operations=[org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@2d026946, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@770b3d32, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@2cce3b32], AFTER_ACTION=[start, compute], BEFORE_ACTION=[start, compute, execute]}
   [junit4]   2> 1026489 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10072_solr&inPlaceMove=true&replica=core_node74
   [junit4]   2> 1026589 DEBUG (simCloudManagerPool-6293-thread-124) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node77":{
   [junit4]   2>     "core":"testBasic_shard5_replica_p76",
   [junit4]   2>     "shard":"shard5",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10072_solr",
   [junit4]   2>     "type":"PULL"}}
   [junit4]   2> 1026701 DEBUG (simCloudManagerPool-6293-thread-126) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testBasic / shard3: {"core_node31":{
   [junit4]   2>     "core":"testBasic_shard3_replica_n31",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "node_name":"127.0.0.1:10074_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard3",
   [junit4]   2>     "collection":"testBasic"}}
   [junit4]   2> 1026702 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10056_solr&inPlaceMove=true&replica=core_node29
   [junit4]   2> 1026803 DEBUG (simCloudManagerPool-6293-thread-132) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node79":{
   [junit4]   2>     "core":"testBasic_shard2_replica_p78",
   [junit4]   2>     "shard":"shard2",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10056_solr",
   [junit4]   2>     "type":"PULL"}}
   [junit4]   2> 1026915 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10059_solr&inPlaceMove=true&replica=core_node32
   [junit4]   2> 1027016 DEBUG (simCloudManagerPool-6293-thread-140) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node81":{
   [junit4]   2>     "core":"testBasic_shard3_replica_n80",
   [junit4]   2>     "shard":"shard3",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10059_solr",
   [junit4]   2>     "type":"NRT"}}
   [junit4]   2> 1027129 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger1 after 100ms
   [junit4]   2> 1027129 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 36348 ms for event id=82ed6c9f3f457T9pw511u6t00f16ou7eh7prv38
   [junit4]   2> 1027230 DEBUG (ScheduledTrigger-6294-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 92
   [junit4]   2> 1027230 DEBUG (ScheduledTrigger-6294-thread-3) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10073_solr, 127.0.0.1:10016_solr, 127.0.0.1:10049_solr, 127.0.0.1:10058_solr, 127.0.0.1:10083_solr]
   [junit4]   2> 1027230 DEBUG (ScheduledTrigger-6294-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"82ed7436034b3T9pw511u6t00f16ou7eh7prv3b",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303301896910003,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303301896910003,
   [junit4]   2>       2303300879474203,
   [junit4]   2>       2303301896392503,
   [junit4]   2>       2303300880739803,
   [junit4]   2>       2303300880180203],
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10073_solr",
   [junit4]   2>       "127.0.0.1:10016_solr",
   [junit4]   2>       "127.0.0.1:10049_solr",
   [junit4]   2>       "127.0.0.1:10058_solr",
   [junit4]   2>       "127.0.0.1:10083_solr"]}}
   [junit4]   2> 1027230 DEBUG (ScheduledTrigger-6294-thread-3) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_trigger1]
   [junit4]   2> 1027232 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"82ed7436034b3T9pw511u6t00f16ou7eh7prv3b",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303301896910003,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303301896910003,
   [junit4]   2>       2303300879474203,
   [junit4]   2>       2303301896392503,
   [junit4]   2>       2303300880739803,
   [junit4]   2>       2303300880180203],
   [junit4]   2>     "_enqueue_time_":2303346507059053,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10073_solr",
   [junit4]   2>       "127.0.0.1:10016_solr",
   [junit4]   2>       "127.0.0.1:10049_solr",
   [junit4]   2>       "127.0.0.1:10058_solr",
   [junit4]   2>       "127.0.0.1:10083_solr"]}}
   [junit4]   2> 1027232 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction -- processing event: {
   [junit4]   2>   "id":"82ed7436034b3T9pw511u6t00f16ou7eh7prv3b",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303301896910003,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303301896910003,
   [junit4]   2>       2303300879474203,
   [junit4]   2>       2303301896392503,
   [junit4]   2>       2303300880739803,
   [junit4]   2>       2303300880180203],
   [junit4]   2>     "_enqueue_time_":2303346507059053,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10073_solr",
   [junit4]   2>       "127.0.0.1:10016_solr",
   [junit4]   2>       "127.0.0.1:10049_solr",
   [junit4]   2>       "127.0.0.1:10058_solr",
   [junit4]   2>       "127.0.0.1:10083_solr"]}} with context properties: {AFTER_ACTION=[start], BEFORE_ACTION=[start, compute]}
   [junit4]   2> 1027273 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10053_solr&inPlaceMove=true&replica=core_node43
   [junit4]   2> 1027306 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10037_solr&inPlaceMove=true&replica=core_node58
   [junit4]   2> 1027323 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10094_solr&inPlaceMove=true&replica=core_node19
   [junit4]   2> 1027332 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10091_solr&inPlaceMove=true&replica=core_node39
   [junit4]   2> 1027333 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2303351614 sessionWrapper.createTime 2303346650057453, this.sessionWrapper.createTime 2303346650057453
   [junit4]   2> 1027333 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction -- processing event: {
   [junit4]   2>   "id":"82ed7436034b3T9pw511u6t00f16ou7eh7prv3b",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303301896910003,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303301896910003,
   [junit4]   2>       2303300879474203,
   [junit4]   2>       2303301896392503,
   [junit4]   2>       2303300880739803,
   [junit4]   2>       2303300880180203],
   [junit4]   2>     "_enqueue_time_":2303346507059053,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10073_solr",
   [junit4]   2>       "127.0.0.1:10016_solr",
   [junit4]   2>       "127.0.0.1:10049_solr",
   [junit4]   2>       "127.0.0.1:10058_solr",
   [junit4]   2>       "127.0.0.1:10083_solr"]}} with context properties: {operations=[org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@52ece38f, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@108b7f38, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@6ec83937, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@46b5d349], AFTER_ACTION=[start, compute], BEFORE_ACTION=[start, compute, execute]}
   [junit4]   2> 1027333 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10053_solr&inPlaceMove=true&replica=core_node43
   [junit4]   2> 1027433 DEBUG (simCloudManagerPool-6293-thread-150) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node83":{
   [junit4]   2>     "core":"testBasic_shard3_replica_p82",
   [junit4]   2>     "shard":"shard3",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10053_solr",
   [junit4]   2>     "type":"PULL"}}
   [junit4]   2> 1027545 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10037_solr&inPlaceMove=true&replica=core_node58
   [junit4]   2> 1027646 DEBUG (simCloudManagerPool-6293-thread-158) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node85":{
   [junit4]   2>     "core":"testBasic_shard4_replica_p84",
   [junit4]   2>     "shard":"shard4",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10037_solr",
   [junit4]   2>     "type":"PULL"}}
   [junit4]   2> 1027758 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10094_solr&inPlaceMove=true&replica=core_node19
   [junit4]   2> 1027858 DEBUG (simCloudManagerPool-6293-thread-167) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node87":{
   [junit4]   2>     "core":"testBasic_shard2_replica_n86",
   [junit4]   2>     "shard":"shard2",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10094_solr",
   [junit4]   2>     "type":"NRT"}}
   [junit4]   2> 1027971 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10091_solr&inPlaceMove=true&replica=core_node39
   [junit4]   2> 1028071 DEBUG (simCloudManagerPool-6293-thread-179) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node89":{
   [junit4]   2>     "core":"testBasic_shard3_replica_t88",
   [junit4]   2>     "shard":"shard3",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10091_solr",
   [junit4]   2>     "type":"TLOG"}}
   [junit4]   2> 1028183 INFO  (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.TestLargeCluster Ready after 92694ms
   [junit4]   2> 1028184 INFO  (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.TestLargeCluster OP COUNTS: {CREATE=1, DELETESTATUS=7, MOVEREPLICA=7, REQUESTSTATUS=7, autoscaling=4, update=9}
   [junit4]   2> 1028184 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger1 after 100ms
   [junit4]   2> 1028184 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 47598 ms for event id=82ed7436034b3T9pw511u6t00f16ou7eh7prv3b
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 82
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10043_solr
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10081_solr
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10084_solr
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10036_solr
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10041_solr
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10014_solr
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10011_solr
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10038_solr
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10067_solr
   [junit4]   2> 1028284 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10042_solr
   [junit4]   2> 1028304 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 82
   [junit4]   2> 1028324 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 82
   [junit4]   2> 1028345 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 82
   [junit4]   2> 1028365 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 82
   [junit4]   2> 1028385 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 82
   [junit4]   2> 1028385 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10043_solr, 127.0.0.1:10067_solr, 127.0.0.1:10041_solr, 127.0.0.1:10042_solr, 127.0.0.1:10038_solr, 127.0.0.1:10081_solr, 127.0.0.1:10014_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10011_solr]
   [junit4]   2> 1028385 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"82eede9cb89f9T9pw511u6t00f16ou7eh7prv3e",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303399178242553,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303399178242553,
   [junit4]   2>       2303399181321553,
   [junit4]   2>       2303399179841453,
   [junit4]   2>       2303399181685453,
   [junit4]   2>       2303399180955453,
   [junit4]   2>       2303399178730503,
   [junit4]   2>       2303399180214853,
   [junit4]   2>       2303399179102403,
   [junit4]   2>       2303399179472153,
   [junit4]   2>       2303399180586853],
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10043_solr",
   [junit4]   2>       "127.0.0.1:10067_solr",
   [junit4]   2>       "127.0.0.1:10041_solr",
   [junit4]   2>       "127.0.0.1:10042_solr",
   [junit4]   2>       "127.0.0.1:10038_solr",
   [junit4]   2>       "127.0.0.1:10081_solr",
   [junit4]   2>       "127.0.0.1:10014_solr",
   [junit4]   2>       "127.0.0.1:10084_solr",
   [junit4]   2>       "127.0.0.1:10036_solr",
   [junit4]   2>       "127.0.0.1:10011_solr"]}}
   [junit4]   2> 1028385 DEBUG (ScheduledTrigger-6294-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_trigger1]
   [junit4]   2> 1028386 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"82eede9cb89f9T9pw511u6t00f16ou7eh7prv3e",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303399178242553,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303399178242553,
   [junit4]   2>       2303399181321553,
   [junit4]   2>       2303399179841453,
   [junit4]   2>       2303399181685453,
   [junit4]   2>       2303399180955453,
   [junit4]   2>       2303399178730503,
   [junit4]   2>       2303399180214853,
   [junit4]   2>       2303399179102403,
   [junit4]   2>       2303399179472153,
   [junit4]   2>       2303399180586853],
   [junit4]   2>     "_enqueue_time_":2303404251537753,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10043_solr",
   [junit4]   2>       "127.0.0.1:10067_solr",
   [junit4]   2>       "127.0.0.1:10041_solr",
   [junit4]   2>       "127.0.0.1:10042_solr",
   [junit4]   2>       "127.0.0.1:10038_solr",
   [junit4]   2>       "127.0.0.1:10081_solr",
   [junit4]   2>       "127.0.0.1:10014_solr",
   [junit4]   2>       "127.0.0.1:10084_solr",
   [junit4]   2>       "127.0.0.1:10036_solr",
   [junit4]   2>       "127.0.0.1:10011_solr"]}}
   [junit4]   2> 1028386 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction -- processing event: {
   [junit4]   2>   "id":"82eede9cb89f9T9pw511u6t00f16ou7eh7prv3e",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303399178242553,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303399178242553,
   [junit4]   2>       2303399181321553,
   [junit4]   2>       2303399179841453,
   [junit4]   2>       2303399181685453,
   [junit4]   2>       2303399180955453,
   [junit4]   2>       2303399178730503,
   [junit4]   2>       2303399180214853,
   [junit4]   2>       2303399179102403,
   [junit4]   2>       2303399179472153,
   [junit4]   2>       2303399180586853],
   [junit4]   2>     "_enqueue_time_":2303404251537753,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10043_solr",
   [junit4]   2>       "127.0.0.1:10067_solr",
   [junit4]   2>       "127.0.0.1:10041_solr",
   [junit4]   2>       "127.0.0.1:10042_solr",
   [junit4]   2>       "127.0.0.1:10038_solr",
   [junit4]   2>       "127.0.0.1:10081_solr",
   [junit4]   2>       "127.0.0.1:10014_solr",
   [junit4]   2>       "127.0.0.1:10084_solr",
   [junit4]   2>       "127.0.0.1:10036_solr",
   [junit4]   2>       "127.0.0.1:10011_solr"]}} with context properties: {AFTER_ACTION=[start], BEFORE_ACTION=[start, compute]}
   [junit4]   2> 1028466 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10034_solr&inPlaceMove=true&replica=core_node28
   [junit4]   2> 1028538 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10031_solr&inPlaceMove=true&replica=core_node48
   [junit4]   2> 1028597 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10097_solr&inPlaceMove=true&replica=core_node2
   [junit4]   2> 1028698 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10002_solr&inPlaceMove=true&replica=core_node6
   [junit4]   2> 1028742 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10078_solr&inPlaceMove=true&replica=core_node17
   [junit4]   2> 1028779 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10075_solr&inPlaceMove=true&replica=core_node34
   [junit4]   2> 1028837 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10050_solr&inPlaceMove=true&replica=core_node4
   [junit4]   2> 1028863 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10024_solr&inPlaceMove=true&replica=core_node62
   [junit4]   2> 1028878 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10008_solr&inPlaceMove=true&replica=core_node68
   [junit4]   2> 1028886 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10062_solr&inPlaceMove=true&replica=core_node24
   [junit4]   2> 1028886 INFO  (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2303429307 sessionWrapper.createTime 2303404344411903, this.sessionWrapper.createTime 2303404344411903
   [junit4]   2> 1028887 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction -- processing event: {
   [junit4]   2>   "id":"82eede9cb89f9T9pw511u6t00f16ou7eh7prv3e",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":2303399178242553,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2303399178242553,
   [junit4]   2>       2303399181321553,
   [junit4]   2>       2303399179841453,
   [junit4]   2>       2303399181685453,
   [junit4]   2>       2303399180955453,
   [junit4]   2>       2303399178730503,
   [junit4]   2>       2303399180214853,
   [junit4]   2>       2303399179102403,
   [junit4]   2>       2303399179472153,
   [junit4]   2>       2303399180586853],
   [junit4]   2>     "_enqueue_time_":2303404251537753,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10043_solr",
   [junit4]   2>       "127.0.0.1:10067_solr",
   [junit4]   2>       "127.0.0.1:10041_solr",
   [junit4]   2>       "127.0.0.1:10042_solr",
   [junit4]   2>       "127.0.0.1:10038_solr",
   [junit4]   2>       "127.0.0.1:10081_solr",
   [junit4]   2>       "127.0.0.1:10014_solr",
   [junit4]   2>       "127.0.0.1:10084_solr",
   [junit4]   2>       "127.0.0.1:10036_solr",
   [junit4]   2>       "127.0.0.1:10011_solr"]}} with context properties: {operations=[org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@145c86ab, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@2060a749, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@2285b260, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@7fa88714, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@f313f94, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@2ace9bd4, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@25573a5f, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@1efc99c6, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@1837d812, org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@4679e0fb], AFTER_ACTION=[start, compute], BEFORE_ACTION=[start, compute, execute]}
   [junit4]   2> 1028887 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10034_solr&inPlaceMove=true&replica=core_node28
   [junit4]   2> 1028988 DEBUG (simCloudManagerPool-6293-thread-167) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node91":{
   [junit4]   2>     "core":"testBasic_shard2_replica_p90",
   [junit4]   2>     "shard":"shard2",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10034_solr",
   [junit4]   2>     "type":"PULL"}}
   [junit4]   2> 1029099 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10031_solr&inPlaceMove=true&replica=core_node48
   [junit4]   2> 1029200 DEBUG (simCloudManagerPool-6293-thread-195) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node93":{
   [junit4]   2>     "core":"testBasic_shard4_replica_n92",
   [junit4]   2>     "shard":"shard4",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10031_solr",
   [junit4]   2>     "type":"NRT"}}
   [junit4]   2> 1029311 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10097_solr&inPlaceMove=true&replica=core_node2
   [junit4]   2> 1029412 DEBUG (simCloudManagerPool-6293-thread-44) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node95":{
   [junit4]   2>     "core":"testBasic_shard1_replica_n94",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10097_solr",
   [junit4]   2>     "type":"NRT"}}
   [junit4]   2> 1029524 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10002_solr&inPlaceMove=true&replica=core_node6
   [junit4]   2> 1029624 DEBUG (simCloudManagerPool-6293-thread-14) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node97":{
   [junit4]   2>     "core":"testBasic_shard1_replica_t96",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10002_solr",
   [junit4]   2>     "type":"TLOG"}}
   [junit4]   2> 1029735 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10078_solr&inPlaceMove=true&replica=core_node17
   [junit4]   2> 1029835 DEBUG (simCloudManagerPool-6293-thread-110) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node99":{
   [junit4]   2>     "core":"testBasic_shard2_replica_n98",
   [junit4]   2>     "shard":"shard2",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10078_solr",
   [junit4]   2>     "type":"NRT"}}
   [junit4]   2> 1029947 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10075_solr&inPlaceMove=true&replica=core_node34
   [junit4]   2> 1030047 DEBUG (simCloudManagerPool-6293-thread-172) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node101":{
   [junit4]   2>     "core":"testBasic_shard3_replica_n100",
   [junit4]   2>     "shard":"shard3",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10075_solr",
   [junit4]   2>     "type":"NRT"}}
   [junit4]   2> 1030158 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10050_solr&inPlaceMove=true&replica=core_node4
   [junit4]   2> 1030258 DEBUG (simCloudManagerPool-6293-thread-155) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node103":{
   [junit4]   2>     "core":"testBasic_shard1_replica_n102",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10050_solr",
   [junit4]   2>     "type":"NRT"}}
   [junit4]   2> 1030370 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10024_solr&inPlaceMove=true&replica=core_node62
   [junit4]   2> 1030470 DEBUG (simCloudManagerPool-6293-thread-129) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node105":{
   [junit4]   2>     "core":"testBasic_shard5_replica_n104",
   [junit4]   2>     "shard":"shard5",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10024_solr",
   [junit4]   2>     "type":"NRT"}}
   [junit4]   2> 1030581 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10008_solr&inPlaceMove=true&replica=core_node68
   [junit4]   2> 1030682 DEBUG (simCloudManagerPool-6293-thread-142) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node107":{
   [junit4]   2>     "core":"testBasic_shard5_replica_t106",
   [junit4]   2>     "shard":"shard5",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10008_solr",
   [junit4]   2>     "type":"TLOG"}}
   [junit4]   2> 1030793 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testBasic&targetNode=127.0.0.1:10062_solr&inPlaceMove=true&replica=core_node24
   [junit4]   2> 1030894 DEBUG (simCloudManagerPool-6293-thread-92) [    ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node109":{
   [junit4]   2>     "core":"testBasic_shard2_replica_t108",
   [junit4]   2>     "shard":"shard2",
   [junit4]   2>     "collection":"testBasic",
   [junit4]   2>     "node_name":"127.0.0.1:10062_solr",
   [junit4]   2>     "type":"TLOG"}}
   [junit4]   2> 1031006 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger1 after 100ms
   [junit4]   2> 1031006 DEBUG (AutoscalingActionExecutor-6295-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 130995 ms for event id=82eede9cb89f9T9pw511u6t00f16ou7eh7prv3e
   [junit4]   2> 1031023 INFO  (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.TestLargeCluster Ready after 21ms
   [junit4]   2> 1031023 INFO  (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.TestLargeCluster ==== Flaky replicas: 38. Additional MOVEREPLICA count: 10
   [junit4]   2> 1031024 INFO  (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testBasic
   [junit4]   2> 1031026 INFO  (TEST-TestLargeCluster.testBasic-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 92
   [junit4]   2> ## Empty nodes: 17
   [junit4]   2> ## Dead nodes: 8
   [junit4]   2> ## 127.0.0.1:10016_solr
   [junit4]   2> ## 127.0.0.1:10026_solr
   [junit4]   2> ## 127.0.0.1:10049_solr
   [junit4]   2> ## 127.0.0.1:10058_solr
   [junit4]   2> ## 127.0.0.1:10073_solr
   [junit4]   2> ## 127.0.0.1:10077_solr
   [junit4]   2> ## 127.0.0.1:10082_solr
   [junit4]   2> ## 127.0.0.1:10083_solr
   [junit4]   2> ## Collections: [.system, testBasic]
   [junit4]   2> ## Max replicas per node: 2
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 76
   [junit4]   2> ## * .system 1
   [junit4]   2> ## - active           1
   [junit4]   2> ## * testBasic 75
   [junit4]   2> ## - active          75
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - CREATE             1
   [junit4]   2> ## - DELETESTATUS      17
   [junit4]   2> ## - MOVEREPLICA       17
   [junit4]   2> ## - REQUESTSTATUS     17
   [junit4]   2> ## - autoscaling        4
   [junit4]   2> ## - update            20
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_lost_trigger1
   [junit4]   2> ## - STARTED         3
   [junit4]   2> ## - SUCCEEDED       3
   [junit4]   2>
   [junit4]   2> 1031074 INFO  (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLost
   [junit4]   2> 1031074 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread   -- deleted /autoscaling/nodeLost/127.0.0.1:10011_solr
   [junit4]   2> 1031074 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10082_solr
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread   -- deleted /autoscaling/nodeLost/127.0.0.1:10014_solr
   [junit4]   2> 1031074 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10049_solr
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread   -- deleted /autoscaling/nodeLost/127.0.0.1:10036_solr
   [junit4]   2> 1031074 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10016_solr
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread   -- deleted /autoscaling/nodeLost/127.0.0.1:10038_solr
   [junit4]   2> 1031074 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10077_solr
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread   -- deleted /autoscaling/nodeLost/127.0.0.1:10041_solr
   [junit4]   2> 1031074 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10083_solr
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread   -- deleted /autoscaling/nodeLost/127.0.0.1:10042_solr
   [junit4]   2> 1031074 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10026_solr
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread   -- deleted /autoscaling/nodeLost/127.0.0.1:10043_solr
   [junit4]   2> 1031074 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10073_solr
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread   -- deleted /autoscaling/nodeLost/127.0.0.1:10067_solr
   [junit4]   2> 1031074 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10058_solr
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread   -- deleted /autoscaling/nodeLost/127.0.0.1:10081_solr
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread   -- deleted /autoscaling/nodeLost/127.0.0.1:10084_solr
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1031074 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 1031075 INFO  (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 1031075 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 1031075 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1031075 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 1031075 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 1031075 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 1031075 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 1031075 DEBUG (TEST-TestLargeCluster.testNodeLost-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1031077 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 1031077 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 1031078 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion -1
   [junit4]   2> 1031078 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 1031078 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10104_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10105_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10036_solr, 127.0.0.1:10084_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10001_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10070_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10087_solr, 127.0.0.1:10041_solr, 127.0.0.1:10009_solr, 127.0.0.1:10000_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10062_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10002_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr]
   [junit4]   2> 1031079 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1031079 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 1031079 DEBUG (ScheduledTrigger-6297-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1031100 DEBUG (ScheduledTrigger-6297-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1031120 DEBUG (ScheduledTrigger-6297-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1031140 DEBUG (ScheduledTrigger-6297-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1031161 DEBUG (ScheduledTrigger-6297-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1031179 DEBUG (simCloudManagerPool-6296-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1031179 DEBUG (simCloudManagerPool-6296-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 1031179 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 1031180 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1031180 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 1031181 DEBUG (ScheduledTrigger-6297-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100
   [junit4]   2> 1031181 DEBUG (simCloudManagerPool-6296-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1031181 DEBUG (simCloudManagerPool-6296-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 1031181 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 1031182 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1031182 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1031182 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 1031204 DEBUG (simCloudManagerPool-6296-thread-5) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1031204 DEBUG (simCloudManagerPool-6296-thread-5) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
   [junit4]   2> 1031204 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 1031205 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger3 - Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10018_solr, 127.0.0.1:10104_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10005_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10105_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10036_solr, 127.0.0.1:10084_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10003_solr, 127.0.0.1:10001_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10070_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_sol

[...truncated too long message...]

   \"node_name\":\"127.0.0.1:10085_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":4,\n    \"INDEX.sizeInGB\":9.5367431640625E-6,\n    \"SEARCHER.searcher.numDocs\":0,\n    \"shard\":\"shard1\",\n    \"collection\":\"testSearchRate\"}}",
   [junit4]   2>       "{\"core_node9\":{\n    \"core\":\"testSearchRate_shard1_replica_n9\",\n    \"SEARCHER.searcher.maxDoc\":0,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    \"INDEX.sizeInBytes\":10240,\n    \"rate\":4.0,\n    \"node_name\":\"127.0.0.1:10063_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":4,\n    \"INDEX.sizeInGB\":9.5367431640625E-6,\n    \"SEARCHER.searcher.numDocs\":0,\n    \"shard\":\"shard1\",\n    \"collection\":\"testSearchRate\"}}"],
   [junit4]   2>     "__start__":3,
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"testSearchRate":{"shard1":40.0}},
   [junit4]   2>     "hotCollections":{"testSearchRate":40.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10001_solr":4.0,
   [junit4]   2>       "127.0.0.1:10085_solr":4.0,
   [junit4]   2>       "127.0.0.1:10090_solr":4.0,
   [junit4]   2>       "127.0.0.1:10089_solr":4.0,
   [junit4]   2>       "127.0.0.1:10086_solr":4.0,
   [junit4]   2>       "127.0.0.1:10063_solr":4.0,
   [junit4]   2>       "127.0.0.1:10092_solr":4.0,
   [junit4]   2>       "127.0.0.1:10093_solr":4.0,
   [junit4]   2>       "127.0.0.1:10060_solr":4.0,
   [junit4]   2>       "127.0.0.1:10023_solr":4.0},
   [junit4]   2>     "_enqueue_time_":2316479535593604,
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  \"first\":\"testSearchRate\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  \"first\":\"testSearchRate\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  \"first\":\"testSearchRate\",\n  \"second\":\"shard1\"}]}}"]}} with context properties: {operations=[org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica@376838a7, org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica@22d1e510, org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica@436ae447], AFTER_ACTION=[compute], BEFORE_ACTION=[compute, execute]}
   [junit4]   2> 222815 DEBUG (AutoscalingActionExecutor-18-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=ADDREPLICA&collection=testSearchRate&shard=shard1&node=127.0.0.1:10056_solr&type=NRT
   [junit4]   2> 222836 INFO  (simCloudManagerPool-16-thread-31) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10056_solr for creating new replica
   [junit4]   2> 222870 DEBUG (AutoscalingActionExecutor-18-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=ADDREPLICA&collection=testSearchRate&shard=shard1&node=127.0.0.1:10059_solr&type=NRT
   [junit4]   2> 222878 INFO  (simCloudManagerPool-16-thread-36) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10059_solr for creating new replica
   [junit4]   2> 222894 DEBUG (AutoscalingActionExecutor-18-thread-1) [    ] o.a.s.c.a.ExecutePlanAction Executing operation: action=ADDREPLICA&collection=testSearchRate&shard=shard1&node=127.0.0.1:10053_solr&type=NRT
   [junit4]   2> 222894 INFO  (simCloudManagerPool-16-thread-41) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10053_solr for creating new replica
   [junit4]   2> 222908 DEBUG (simCloudManagerPool-16-thread-46) [    ] o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{"core_node8":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10092_solr",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 222908 DEBUG (AutoscalingActionExecutor-18-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: search_rate_trigger after 100ms
   [junit4]   2> 222908 DEBUG (AutoscalingActionExecutor-18-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 5781 ms for event id=83ad1fa2c1de2T9pw511u6t00f16ou7eh7prpkr
   [junit4]   2> 222947 INFO  (TEST-TestLargeCluster.testSearchRate-seed#[1961A9D5677685CF]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSearchRate
   [junit4]   2> 222948 INFO  (TEST-TestLargeCluster.testSearchRate-seed#[1961A9D5677685CF]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 100
   [junit4]   2> ## Empty nodes: 77
   [junit4]   2> ## Dead nodes: 0
   [junit4]   2> ## Collections: [.system, testSearchRate]
   [junit4]   2> ## Max replicas per node: 2
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 24
   [junit4]   2> ## * .system 1
   [junit4]   2> ## - active           1
   [junit4]   2> ## * testSearchRate 23
   [junit4]   2> ## - active          23
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - ADDREPLICA         3
   [junit4]   2> ## - CREATE             1
   [junit4]   2> ## - DELETESTATUS       3
   [junit4]   2> ## - REQUESTSTATUS      3
   [junit4]   2> ## - autoscaling        4
   [junit4]   2> ## - update             2
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: search_rate_trigger
   [junit4]   2> ## - STARTED         1
   [junit4]   2> ## - SUCCEEDED       1
   [junit4]   2>
   [junit4] OK      1.74s J1 | TestLargeCluster.testSearchRate
   [junit4]   2> 222949 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 222949 DEBUG (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 222949 DEBUG (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 222949 DEBUG (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 222949 DEBUG (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 222949 DEBUG (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 222949 DEBUG (SUITE-TestLargeCluster-seed#[1961A9D5677685CF]-worker) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestLargeCluster_1961A9D5677685CF-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=dua, timezone=Asia/Qatar
   [junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=8,threads=1,free=312319040,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestLargeCluster]
   [junit4] Completed [3/5 (3!)] on J1 in 222.18s, 4 tests, 1 failure <<< FAILURES!

[...truncated 27 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/common-build.xml:1568: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/common-build.xml:1092: There were test failures: 5 suites, 20 tests, 3 failures [seed: 1961A9D5677685CF]

Total time: 4 minutes 16 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   3/5 failed: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=35a7e95bce54d53820ce3e0b1e2966f609a1c1d2, workspace=/var/lib/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #50
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-BadApples-7.x-Linux (32bit/jdk1.8.0_172) - Build # 52 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-7.x-Linux/52/
Java: 32bit/jdk1.8.0_172 -client -XX:+UseSerialGC

7 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.HttpPartitionTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [ZkCollectionTerms] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:39)  at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1505)  at org.apache.solr.cloud.ZkController.unregister(ZkController.java:1524)  at org.apache.solr.cloud.ZkController.unregister(ZkController.java:1518)  at org.apache.solr.core.CoreContainer.unload(CoreContainer.java:1516)  at org.apache.solr.cloud.ZkController$UnloadCoreOnDeletedWatcher.onStateChanged(ZkController.java:2746)  at org.apache.solr.common.cloud.ZkStateReader$Notification.run(ZkStateReader.java:1642)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [ZkCollectionTerms]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:39)
        at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1505)
        at org.apache.solr.cloud.ZkController.unregister(ZkController.java:1524)
        at org.apache.solr.cloud.ZkController.unregister(ZkController.java:1518)
        at org.apache.solr.core.CoreContainer.unload(CoreContainer.java:1516)
        at org.apache.solr.cloud.ZkController$UnloadCoreOnDeletedWatcher.onStateChanged(ZkController.java:2746)
        at org.apache.solr.common.cloud.ZkStateReader$Notification.run(ZkStateReader.java:1642)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([58C3B8EE75A72DF1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:304)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testBasic

Error Message:
did not finish processing all events in time: started=3, finished=2

Stack Trace:
java.lang.AssertionError: did not finish processing all events in time: started=3, finished=2
        at __randomizedtesting.SeedInfo.seed([58C3B8EE75A72DF1:F339A5FBAA7BABDF]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testBasic(TestLargeCluster.java:253)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testBasic

Error Message:
did not finish processing all events in time: started=4, finished=3

Stack Trace:
java.lang.AssertionError: did not finish processing all events in time: started=4, finished=3
        at __randomizedtesting.SeedInfo.seed([58C3B8EE75A72DF1:F339A5FBAA7BABDF]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testBasic(TestLargeCluster.java:253)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
did not finish processing all events in time: started=5, finished=4

Stack Trace:
java.lang.AssertionError: did not finish processing all events in time: started=5, finished=4
        at __randomizedtesting.SeedInfo.seed([58C3B8EE75A72DF1:E7D67610F64D4877]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.doTestNodeLost(TestLargeCluster.java:522)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost(TestLargeCluster.java:375)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
did not finish processing all events in time: started=5, finished=4

Stack Trace:
java.lang.AssertionError: did not finish processing all events in time: started=5, finished=4
        at __randomizedtesting.SeedInfo.seed([58C3B8EE75A72DF1:E7D67610F64D4877]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.doTestNodeLost(TestLargeCluster.java:522)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost(TestLargeCluster.java:375)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
did not finish processing all events in time: started=5, finished=4

Stack Trace:
java.lang.AssertionError: did not finish processing all events in time: started=5, finished=4
        at __randomizedtesting.SeedInfo.seed([58C3B8EE75A72DF1:E7D67610F64D4877]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.doTestNodeLost(TestLargeCluster.java:522)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost(TestLargeCluster.java:375)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
did not finish processing all events in time: started=5, finished=4

Stack Trace:
java.lang.AssertionError: did not finish processing all events in time: started=5, finished=4
        at __randomizedtesting.SeedInfo.seed([58C3B8EE75A72DF1:E7D67610F64D4877]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.doTestNodeLost(TestLargeCluster.java:522)
        at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost(TestLargeCluster.java:375)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12516 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/init-core-data-001
   [junit4]   2> 65500 WARN  (SUITE-HttpPartitionTest-seed#[58C3B8EE75A72DF1]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=7 numCloses=7
   [junit4]   2> 65501 INFO  (SUITE-HttpPartitionTest-seed#[58C3B8EE75A72DF1]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 65502 INFO  (SUITE-HttpPartitionTest-seed#[58C3B8EE75A72DF1]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 65502 INFO  (SUITE-HttpPartitionTest-seed#[58C3B8EE75A72DF1]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 65502 INFO  (SUITE-HttpPartitionTest-seed#[58C3B8EE75A72DF1]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /jr_i/k
   [junit4]   2> 65504 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 65504 INFO  (Thread-134) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 65504 INFO  (Thread-134) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 65506 ERROR (Thread-134) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 65604 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.ZkTestServer start zk server on port:33613
   [junit4]   2> 65607 INFO  (zkConnectionManagerCallback-142-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 65618 INFO  (zkConnectionManagerCallback-144-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 65624 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 65626 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 65633 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 65635 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 65636 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 65637 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 65639 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 65640 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 65642 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 65643 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 65644 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 65645 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 65726 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 65727 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 65727 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 65727 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 65727 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f39877{/jr_i/k,null,AVAILABLE}
   [junit4]   2> 65729 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3a1623{HTTP/1.1,[http/1.1]}{127.0.0.1:46005}
   [junit4]   2> 65729 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.Server Started @65766ms
   [junit4]   2> 65729 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/tempDir-001/control/data, hostContext=/jr_i/k, hostPort=44301, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/control-001/cores}
   [junit4]   2> 65729 ERROR (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 65729 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 65729 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.5.0
   [junit4]   2> 65729 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 65730 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 65730 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-06-18T05:38:04.373Z
   [junit4]   2> 65731 INFO  (zkConnectionManagerCallback-146-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 65733 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 65733 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/control-001/solr.xml
   [junit4]   2> 65740 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 65740 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 65744 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 67014 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33613/solr
   [junit4]   2> 67017 INFO  (zkConnectionManagerCallback-150-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 67021 INFO  (zkConnectionManagerCallback-152-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 67088 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 67088 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44301_jr_i%2Fk
   [junit4]   2> 67089 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.c.Overseer Overseer (id=72220214795173892-127.0.0.1:44301_jr_i%2Fk-n_0000000000) starting
   [junit4]   2> 67094 INFO  (zkConnectionManagerCallback-159-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 67096 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33613/solr ready
   [junit4]   2> 67098 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44301_jr_i%2Fk
   [junit4]   2> 67099 INFO  (zkCallback-151-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 67099 INFO  (zkCallback-158-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 67117 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 67141 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 67151 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 67152 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 67153 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/control-001/cores
   [junit4]   2> 67174 INFO  (zkConnectionManagerCallback-165-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 67175 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 67176 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33613/solr ready
   [junit4]   2> 67177 INFO  (SocketProxy-Acceptor-44301) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=42126,localport=44301], receiveBufferSize:531000
   [junit4]   2> 67181 INFO  (SocketProxy-Acceptor-44301) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=46005,localport=42432], receiveBufferSize=530904
   [junit4]   2> 67182 INFO  (qtp32117015-617) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44301_jr_i%252Fk&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 67184 INFO  (OverseerThreadFactory-319-thread-1) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 67291 INFO  (SocketProxy-Acceptor-44301) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=42130,localport=44301], receiveBufferSize:531000
   [junit4]   2> 67292 INFO  (SocketProxy-Acceptor-44301) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=46005,localport=42436], receiveBufferSize=530904
   [junit4]   2> 67294 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 67294 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 68328 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
   [junit4]   2> 68373 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 68508 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 68532 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 68532 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 68532 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 68532 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 68536 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@cc3a98
   [junit4]   2> 68538 WARN  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 68596 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 68596 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 68598 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 68598 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 68599 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=78.1484375, floorSegmentMB=0.9306640625, forceMergeDeletesPctAllowed=7.338954919323935, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, reclaimDeletesWeight=0.17701508674919264
   [junit4]   2> 68600 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@805f8c[control_collection_shard1_replica_n1] main]
   [junit4]   2> 68601 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 68602 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 68602 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 68604 INFO  (searcherExecutor-324-thread-1-processing-n:127.0.0.1:44301_jr_i%2Fk x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@805f8c[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 68604 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1603587578000310272
   [junit4]   2> 68610 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 68613 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 68613 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 68613 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44301/jr_i/k/control_collection_shard1_replica_n1/
   [junit4]   2> 68613 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 68613 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:44301/jr_i/k/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 68613 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 68615 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44301/jr_i/k/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 68768 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 68791 INFO  (qtp32117015-612) [n:127.0.0.1:44301_jr_i%2Fk c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1497
   [junit4]   2> 68794 INFO  (qtp32117015-617) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 68872 INFO  (zkCallback-151-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 69186 INFO  (OverseerCollectionConfigSetProcessor-72220214795173892-127.0.0.1:44301_jr_i%2Fk-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 69795 INFO  (qtp32117015-617) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44301_jr_i%252Fk&wt=javabin&version=2} status=0 QTime=2613
   [junit4]   2> 69801 INFO  (zkConnectionManagerCallback-170-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 69803 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 69804 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33613/solr ready
   [junit4]   2> 69804 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 69804 INFO  (SocketProxy-Acceptor-44301) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=42146,localport=44301], receiveBufferSize:531000
   [junit4]   2> 69807 INFO  (SocketProxy-Acceptor-44301) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=46005,localport=42452], receiveBufferSize=530904
   [junit4]   2> 69808 INFO  (qtp32117015-613) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 69811 INFO  (OverseerThreadFactory-319-thread-2) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 69812 WARN  (OverseerThreadFactory-319-thread-2) [    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 70017 INFO  (qtp32117015-613) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 70018 INFO  (qtp32117015-613) [n:127.0.0.1:44301_jr_i%2Fk    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=210
   [junit4]   2> 70101 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-1-001 of type TLOG
   [junit4]   2> 70101 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 70102 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 70102 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 70102 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 70103 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f8d51c{/jr_i/k,null,AVAILABLE}
   [junit4]   2> 70103 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@f03172{HTTP/1.1,[http/1.1]}{127.0.0.1:40193}
   [junit4]   2> 70104 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.Server Started @70141ms
   [junit4]   2> 70104 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/jr_i/k, hostPort=33293, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-1-001/cores}
   [junit4]   2> 70104 ERROR (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 70104 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 70104 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.5.0
   [junit4]   2> 70104 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 70104 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 70104 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-06-18T05:38:08.747Z
   [junit4]   2> 70106 INFO  (zkConnectionManagerCallback-172-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 70107 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 70107 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-1-001/solr.xml
   [junit4]   2> 70114 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 70114 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 70115 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 71724 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33613/solr
   [junit4]   2> 71726 INFO  (zkConnectionManagerCallback-176-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 71730 INFO  (zkConnectionManagerCallback-178-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 71735 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 71736 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 71737 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 71738 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33293_jr_i%2Fk
   [junit4]   2> 71738 INFO  (zkCallback-158-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 71738 INFO  (zkCallback-151-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 71739 INFO  (zkCallback-169-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 71739 INFO  (zkCallback-177-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 71756 INFO  (zkConnectionManagerCallback-185-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 71757 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 71759 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33613/solr ready
   [junit4]   2> 71759 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 71781 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 71803 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 71803 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 71805 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-1-001/cores
   [junit4]   2> 71813 INFO  (OverseerCollectionConfigSetProcessor-72220214795173892-127.0.0.1:44301_jr_i%2Fk-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 71830 INFO  (SocketProxy-Acceptor-33293) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=47434,localport=33293], receiveBufferSize:531000
   [junit4]   2> 71831 INFO  (SocketProxy-Acceptor-33293) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40193,localport=57746], receiveBufferSize=530904
   [junit4]   2> 71832 INFO  (qtp20324729-681) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:33293_jr_i%252Fk&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 71836 INFO  (OverseerThreadFactory-319-thread-3) [ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33293_jr_i%2Fk for creating new replica
   [junit4]   2> 71839 INFO  (SocketProxy-Acceptor-33293) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=47438,localport=33293], receiveBufferSize:531000
   [junit4]   2> 71839 INFO  (SocketProxy-Acceptor-33293) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40193,localport=57750], receiveBufferSize=530904
   [junit4]   2> 71841 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk    x:collection1_shard2_replica_t41] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 72884 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
   [junit4]   2> 72908 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema [collection1_shard2_replica_t41] Schema name=test
   [junit4]   2> 73021 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 73039 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t41' using configuration from collection collection1, trusted=true
   [junit4]   2> 73040 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_t41' (registry 'solr.core.collection1.shard2.replica_t41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 73040 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 73040 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [[collection1_shard2_replica_t41] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-1-001/cores/collection1_shard2_replica_t41], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-1-001/cores/collection1_shard2_replica_t41/data/]
   [junit4]   2> 73043 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@b46e7c
   [junit4]   2> 73044 WARN  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 73098 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 73098 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 73100 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 73100 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 73101 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=78.1484375, floorSegmentMB=0.9306640625, forceMergeDeletesPctAllowed=7.338954919323935, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, reclaimDeletesWeight=0.17701508674919264
   [junit4]   2> 73102 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.SolrIndexSearcher Opening [Searcher@19d4fee[collection1_shard2_replica_t41] main]
   [junit4]   2> 73103 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 73103 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 73104 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 73105 INFO  (searcherExecutor-338-thread-1-processing-n:127.0.0.1:33293_jr_i%2Fk x:collection1_shard2_replica_t41 c:collection1 s:shard2) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [collection1_shard2_replica_t41] Registered new searcher Searcher@19d4fee[collection1_shard2_replica_t41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 73105 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1603587582719950848
   [junit4]   2> 73110 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node42=0}, version=0}
   [junit4]   2> 73112 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 73112 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 73112 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33293/jr_i/k/collection1_shard2_replica_t41/
   [junit4]   2> 73112 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 73113 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy http://127.0.0.1:33293/jr_i/k/collection1_shard2_replica_t41/ has no replicas
   [junit4]   2> 73113 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 73113 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ZkController collection1_shard2_replica_t41 stopping background replication from leader
   [junit4]   2> 73115 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33293/jr_i/k/collection1_shard2_replica_t41/ shard2
   [junit4]   2> 73217 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 73219 INFO  (qtp20324729-676) [n:127.0.0.1:33293_jr_i%2Fk c:collection1 s:shard2  x:collection1_shard2_replica_t41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1378
   [junit4]   2> 73222 INFO  (qtp20324729-681) [n:127.0.0.1:33293_jr_i%2Fk c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:33293_jr_i%252Fk&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2} status=0 QTime=1389
   [junit4]   2> 73311 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-2-001 of type TLOG
   [junit4]   2> 73311 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 73312 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 73312 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 73312 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 73313 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@20a93e{/jr_i/k,null,AVAILABLE}
   [junit4]   2> 73314 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1508a70{HTTP/1.1,[http/1.1]}{127.0.0.1:40207}
   [junit4]   2> 73314 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.e.j.s.Server Started @73351ms
   [junit4]   2> 73314 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/tempDir-001/jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/jr_i/k, hostPort=34749, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-2-001/cores}
   [junit4]   2> 73314 ERROR (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 73314 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 73314 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.5.0
   [junit4]   2> 73315 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 73315 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 73315 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-06-18T05:38:11.958Z
   [junit4]   2> 73316 INFO  (zkConnectionManagerCallback-188-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 73318 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 73318 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-2-001/solr.xml
   [junit4]   2> 73319 INFO  (zkCallback-177-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 73322 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 73322 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 73323 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 73836 INFO  (OverseerCollectionConfigSetProcessor-72220214795173892-127.0.0.1:44301_jr_i%2Fk-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 75047 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33613/solr
   [junit4]   2> 75049 INFO  (zkConnectionManagerCallback-192-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 75056 INFO  (zkConnectionManagerCallback-194-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 75062 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 75064 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 75066 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 75066 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34749_jr_i%2Fk
   [junit4]   2> 75067 INFO  (zkCallback-169-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 75067 INFO  (zkCallback-158-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 75067 INFO  (zkCallback-177-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 75067 INFO  (zkCallback-151-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 75069 INFO  (zkCallback-184-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 75069 INFO  (zkCallback-193-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 75097 INFO  (zkConnectionManagerCallback-201-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 75098 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 75099 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33613/solr ready
   [junit4]   2> 75100 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 75121 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 75136 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 75136 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 75138 INFO  (TEST-HttpPartitionTest.test-seed#[58C3B8EE75A72DF1]) [n:127.0.0.1:34749_jr_i%2Fk    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-2-001/cores
   [junit4]   2> 75175 INFO  (qtp20324729-675) [n:127.0.0.1:33293_jr_i%2Fk    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34749_jr_i%252Fk&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 75179 INFO  (OverseerThreadFactory-319-thread-4) [ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:34749_jr_i%2Fk for creating new replica
   [junit4]   2> 75181 INFO  (SocketProxy-Acceptor-34749) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=52258,localport=34749], receiveBufferSize:531000
   [junit4]   2> 75182 INFO  (SocketProxy-Acceptor-34749) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40207,localport=46150], receiveBufferSize=530904
   [junit4]   2> 75184 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk    x:collection1_shard1_replica_t43] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 75288 INFO  (zkCallback-177-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 76241 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.5.0
   [junit4]   2> 76260 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.s.IndexSchema [collection1_shard1_replica_t43] Schema name=test
   [junit4]   2> 76421 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 76446 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t43' using configuration from collection collection1, trusted=true
   [junit4]   2> 76447 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t43' (registry 'solr.core.collection1.shard1.replica_t43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ccf953
   [junit4]   2> 76447 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 76447 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.SolrCore [[collection1_shard1_replica_t43] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-2-001/cores/collection1_shard1_replica_t43], dataDir=[/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_58C3B8EE75A72DF1-001/shard-2-001/cores/collection1_shard1_replica_t43/data/]
   [junit4]   2> 76450 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@ef686e
   [junit4]   2> 76451 WARN  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 76511 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 76511 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 76513 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 76513 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 76515 INFO  (qtp18928032-726) [n:127.0.0.1:34749_jr_i%2Fk c:collection1 s:shard1  x:collection1_shard1_replica_t43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=78.1484375, floorSegmentMB=0.9306640

[...truncated too long message...]

lr&inPlaceMove=true&replica=core_node124
   [junit4]   2> 543689 INFO  (AutoscalingActionExecutor-259-thread-1) [    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2508894149 sessionWrapper.createTime 2508845123171295, this.sessionWrapper.createTime 2508845123171295
   [junit4]   2> 543689 WARN  (AutoscalingActionExecutor-259-thread-1) [    ] o.a.s.c.a.SystemLogListener Exception sending event to collection .system
   [junit4]   2> java.io.IOException: java.lang.InterruptedException
   [junit4]   2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:632) ~[test/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:118) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:790) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:311) ~[java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> Caused by: java.lang.InterruptedException
   [junit4]   2> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_172]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:630) ~[test/:?]
   [junit4]   2> ... 9 more
   [junit4]   2> 543690 WARN  (AutoscalingActionExecutor-259-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> java.lang.Exception: Error executing action: compute for trigger event: {
   [junit4]   2>   "id":"8e9ba6e00120fT7xk3ke4qhab2gnoaz7b7n57mw",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":2508786732372495,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2508786732372495,
   [junit4]   2>       2508786736050295,
   [junit4]   2>       2508786734790295,
   [junit4]   2>       2508786733066795,
   [junit4]   2>       2508786733943445,
   [junit4]   2>       2508786735633395,
   [junit4]   2>       2508786733509945,
   [junit4]   2>       2508786736471995,
   [junit4]   2>       2508786735212945,
   [junit4]   2>       2508786734365395,
   [junit4]   2>       2508786736894195],
   [junit4]   2>     "_enqueue_time_":2508844916396495,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10282_solr",
   [junit4]   2>       "127.0.0.1:10286_solr",
   [junit4]   2>       "127.0.0.1:10278_solr",
   [junit4]   2>       "127.0.0.1:10279_solr",
   [junit4]   2>       "127.0.0.1:10284_solr",
   [junit4]   2>       "127.0.0.1:10287_solr",
   [junit4]   2>       "127.0.0.1:10281_solr",
   [junit4]   2>       "127.0.0.1:10280_solr",
   [junit4]   2>       "127.0.0.1:10288_solr",
   [junit4]   2>       "127.0.0.1:10285_solr",
   [junit4]   2>       "127.0.0.1:10283_solr"]}}
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:312) ~[java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unexpected exception while processing event: {
   [junit4]   2>   "id":"8e9ba6e00120fT7xk3ke4qhab2gnoaz7b7n57mw",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":2508786732372495,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[
   [junit4]   2>       2508786732372495,
   [junit4]   2>       2508786736050295,
   [junit4]   2>       2508786734790295,
   [junit4]   2>       2508786733066795,
   [junit4]   2>       2508786733943445,
   [junit4]   2>       2508786735633395,
   [junit4]   2>       2508786733509945,
   [junit4]   2>       2508786736471995,
   [junit4]   2>       2508786735212945,
   [junit4]   2>       2508786734365395,
   [junit4]   2>       2508786736894195],
   [junit4]   2>     "_enqueue_time_":2508844916396495,
   [junit4]   2>     "nodeNames":[
   [junit4]   2>       "127.0.0.1:10282_solr",
   [junit4]   2>       "127.0.0.1:10286_solr",
   [junit4]   2>       "127.0.0.1:10278_solr",
   [junit4]   2>       "127.0.0.1:10279_solr",
   [junit4]   2>       "127.0.0.1:10284_solr",
   [junit4]   2>       "127.0.0.1:10287_solr",
   [junit4]   2>       "127.0.0.1:10281_solr",
   [junit4]   2>       "127.0.0.1:10280_solr",
   [junit4]   2>       "127.0.0.1:10288_solr",
   [junit4]   2>       "127.0.0.1:10285_solr",
   [junit4]   2>       "127.0.0.1:10283_solr"]}}
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:156) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309) ~[java/:?]
   [junit4]   2> ... 6 more
   [junit4]   2> Caused by: java.lang.InterruptedException: stopping - thread was interrupted
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:108) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309) ~[java/:?]
   [junit4]   2> ... 6 more
   [junit4]   2> 543690 DEBUG (AutoscalingActionExecutor-259-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 49178 ms for event id=8e9ba6e00120fT7xk3ke4qhab2gnoaz7b7n57mw
   [junit4]   2> 543690 DEBUG (SUITE-TestLargeCluster-seed#[58C3B8EE75A72DF1]-worker) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 543690 DEBUG (SUITE-TestLargeCluster-seed#[58C3B8EE75A72DF1]-worker) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 543690 DEBUG (SUITE-TestLargeCluster-seed#[58C3B8EE75A72DF1]-worker) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestLargeCluster_58C3B8EE75A72DF1-002
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1597, maxMBSortInHeap=6.380549246985257, sim=RandomSimilarity(queryNorm=false): {}, locale=lv-LV, timezone=Asia/Gaza
   [junit4]   2> NOTE: Linux 4.13.0-41-generic i386/Oracle Corporation 1.8.0_172 (32-bit)/cpus=8,threads=1,free=115361688,total=284688384
   [junit4]   2> NOTE: All tests run in this JVM: [HttpPartitionTest, TestLargeCluster, TestLargeCluster]
   [junit4] Completed [10/10 (5!)] on J0 in 229.27s, 4 tests, 1 failure <<< FAILURES!

[...truncated 15 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/common-build.xml:1568: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/common-build.xml:1092: There were test failures: 10 suites, 25 tests, 5 failures [seed: 58C3B8EE75A72DF1]

Total time: 9 minutes 6 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.HttpPartitionTest
[repro]   5/5 failed: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=e5407c5a9710247e5f728aae36224a245a51f0b1, workspace=/var/lib/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #51
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS-EA] Lucene-Solr-BadApples-7.x-Linux (64bit/jdk-11-ea+14) - Build # 53 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-7.x-Linux/53/
Java: 64bit/jdk-11-ea+14 -XX:-UseCompressedOops -XX:+UseSerialGC

9 tests failed.
FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:43183/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:42907/solr/MoveReplicaHDFSTest_failed_coll_true]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:43183/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:42907/solr/MoveReplicaHDFSTest_failed_coll_true]
        at __randomizedtesting.SeedInfo.seed([A5F59192A7805BDB:F38426010538E0B]:0)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:993)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:288)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:43183/solr/MoveReplicaHDFSTest_failed_coll_true: no servers hosting shard: shard2
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:436)
        ... 45 more


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds

Error Message:
failed to create testMixedBounds_collection Live Nodes: [127.0.0.1:10003_solr, 127.0.0.1:10002_solr] Last available state: DocCollection(testMixedBounds_collection//clusterstate.json/26)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testMixedBounds_collection_shard2_replica_n3",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node4":{           "core":"testMixedBounds_collection_shard2_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"0-7fffffff",       "state":"active"},     "shard1":{       "stateTimestamp":"1529485711559648900",       "replicas":{         "core_node1":{           "core":"testMixedBounds_collection_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":495,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":257740,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.4003908038139343E-4,           "SEARCHER.searcher.numDocs":495},         "core_node2":{           "core":"testMixedBounds_collection_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":495,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":257740,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.4003908038139343E-4,           "SEARCHER.searcher.numDocs":495}},       "range":"80000000-ffffffff",       "state":"inactive"},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1529485711560061900",       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node10":{           "leader":"true",           "core":"testMixedBounds_collection_shard1_1_replica1",           "SEARCHER.searcher.maxDoc":247,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":133740,           "node_name":"127.0.0.1:10002_solr",           "base_url":"http://127.0.0.1:10002/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.245550811290741E-4,           "SEARCHER.searcher.numDocs":247},         "core_node9":{           "core":"testMixedBounds_collection_shard1_1_replica0",           "SEARCHER.searcher.maxDoc":247,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":133740,           "node_name":"127.0.0.1:10003_solr",           "base_url":"http://127.0.0.1:10003/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.245550811290741E-4,           "SEARCHER.searcher.numDocs":247}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1529485711559981200",       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node7":{           "leader":"true",           "core":"testMixedBounds_collection_shard1_0_replica0",           "SEARCHER.searcher.maxDoc":248,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":144480,           "node_name":"127.0.0.1:10003_solr",           "base_url":"http://127.0.0.1:10003/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.3455748558044434E-4,           "SEARCHER.searcher.numDocs":248},         "core_node8":{           "core":"testMixedBounds_collection_shard1_0_replica1",           "SEARCHER.searcher.maxDoc":248,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":144480,           "node_name":"127.0.0.1:10002_solr",           "base_url":"http://127.0.0.1:10002/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.3455748558044434E-4,           "SEARCHER.searcher.numDocs":248}}}}}

Stack Trace:
java.lang.AssertionError: failed to create testMixedBounds_collection
Live Nodes: [127.0.0.1:10003_solr, 127.0.0.1:10002_solr]
Last available state: DocCollection(testMixedBounds_collection//clusterstate.json/26)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testMixedBounds_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10003_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node4":{
          "core":"testMixedBounds_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10002_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1529485711559648900",
      "replicas":{
        "core_node1":{
          "core":"testMixedBounds_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10003_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495},
        "core_node2":{
          "core":"testMixedBounds_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10002_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1529485711560061900",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10002_solr",
          "base_url":"http://127.0.0.1:10002/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247},
        "core_node9":{
          "core":"testMixedBounds_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10003_solr",
          "base_url":"http://127.0.0.1:10003/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1529485711559981200",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10003_solr",
          "base_url":"http://127.0.0.1:10003/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248},
        "core_node8":{
          "core":"testMixedBounds_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10002_solr",
          "base_url":"http://127.0.0.1:10002/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248}}}}}
        at __randomizedtesting.SeedInfo.seed([A5F59192A7805BDB:AF762E3FEA3B5081]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:69)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds(IndexSizeTriggerTest.java:561)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.util.concurrent.TimeoutException: last state: DocCollection(testMixedBounds_collection//clusterstate.json/26)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testMixedBounds_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10003_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node4":{
          "core":"testMixedBounds_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10002_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1529485711559648900",
      "replicas":{
        "core_node1":{
          "core":"testMixedBounds_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10003_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495},
        "core_node2":{
          "core":"testMixedBounds_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10002_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1529485711560061900",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10002_solr",
          "base_url":"http://127.0.0.1:10002/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247},
        "core_node9":{
          "core":"testMixedBounds_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10003_solr",
          "base_url":"http://127.0.0.1:10003/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1529485711559981200",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10003_solr",
          "base_url":"http://127.0.0.1:10003/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248},
        "core_node8":{
          "core":"testMixedBounds_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10002_solr",
          "base_url":"http://127.0.0.1:10002/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248}}}}}
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:63)
        ... 39 more


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: DocCollection(testSplitIntegration_collection//clusterstate.json/66)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testSplitIntegration_collection_shard2_replica_n3",           "leader":"true",           "SEARCHER.searcher.maxDoc":11,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":15740,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.4659017324447632E-5,           "SEARCHER.searcher.numDocs":11},         "core_node4":{           "core":"testSplitIntegration_collection_shard2_replica_n4",           "SEARCHER.searcher.maxDoc":11,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":15740,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.4659017324447632E-5,           "SEARCHER.searcher.numDocs":11}},       "range":"0-7fffffff",       "state":"active"},     "shard1":{       "stateTimestamp":"1529485827755713700",       "replicas":{         "core_node1":{           "core":"testSplitIntegration_collection_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14},         "core_node2":{           "core":"testSplitIntegration_collection_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14}},       "range":"80000000-ffffffff",       "state":"inactive"},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1529485827756040850",       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node10":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_1_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10002_solr",           "base_url":"http://127.0.0.1:10002/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7},         "core_node9":{           "core":"testSplitIntegration_collection_shard1_1_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10003_solr",           "base_url":"http://127.0.0.1:10003/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1529485827755980050",       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node7":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_0_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10003_solr",           "base_url":"http://127.0.0.1:10003/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7},         "core_node8":{           "core":"testSplitIntegration_collection_shard1_0_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10002_solr",           "base_url":"http://127.0.0.1:10002/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7}}}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: DocCollection(testSplitIntegration_collection//clusterstate.json/66)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testSplitIntegration_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":11,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":15740,
          "node_name":"127.0.0.1:10003_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.4659017324447632E-5,
          "SEARCHER.searcher.numDocs":11},
        "core_node4":{
          "core":"testSplitIntegration_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":11,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":15740,
          "node_name":"127.0.0.1:10002_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.4659017324447632E-5,
          "SEARCHER.searcher.numDocs":11}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1529485827755713700",
      "replicas":{
        "core_node1":{
          "core":"testSplitIntegration_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10003_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14},
        "core_node2":{
          "core":"testSplitIntegration_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10002_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1529485827756040850",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10002_solr",
          "base_url":"http://127.0.0.1:10002/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node9":{
          "core":"testSplitIntegration_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10003_solr",
          "base_url":"http://127.0.0.1:10003/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1529485827755980050",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10003_solr",
          "base_url":"http://127.0.0.1:10003/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node8":{
          "core":"testSplitIntegration_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10002_solr",
          "base_url":"http://127.0.0.1:10002/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7}}}}}
        at __randomizedtesting.SeedInfo.seed([A5F59192A7805BDB:9C7B28D2887F9225]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration(IndexSizeTriggerTest.java:301)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:832)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds

Error Message:
failed to create testMixedBounds_collection Live Nodes: [127.0.0.1:10004_solr, 127.0.0.1:10005_solr] Last available state: DocCollection(testMixedBounds_collection//clusterstate.json/27)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testMixedBounds_collection_shard2_replica_n3",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10005_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node4":{           "core":"testMixedBounds_collection_shard2_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10004_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"0-7fffffff",       "state":"active"},     "shard1":{       "stateTimestamp":"1529485957954308300",       "replicas":{         "core_node1":{           "core":"testMixedBounds_collection_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":495,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":257740,           "node_name":"127.0.0.1:10005_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.4003908038139343E-4,           "SEARCHER.searcher.numDocs":495},         "core_node2":{           "core":"testMixedBounds_collection_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":495,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":257740,           "node_name":"127.0.0.1:10004_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.4003908038139343E-4,           "SEARCHER.searcher.numDocs":495}},       "range":"80000000-ffffffff",       "state":"inactive"},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1529485957954629200",       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node10":{           "leader":"true",           "core":"testMixedBounds_collection_shard1_1_replica1",           "SEARCHER.searcher.maxDoc":247,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":133740,           "node_name":"127.0.0.1:10005_solr",           "base_url":"http://127.0.0.1:10005/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.245550811290741E-4,           "SEARCHER.searcher.numDocs":247},         "core_node9":{           "core":"testMixedBounds_collection_shard1_1_replica0",           "SEARCHER.searcher.maxDoc":247,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":133740,           "node_name":"127.0.0.1:10004_solr",           "base_url":"http://127.0.0.1:10004/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.245550811290741E-4,           "SEARCHER.searcher.numDocs":247}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1529485957954567800",       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node7":{           "leader":"true",           "core":"testMixedBounds_collection_shard1_0_replica0",           "SEARCHER.searcher.maxDoc":248,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":144480,           "node_name":"127.0.0.1:10004_solr",           "base_url":"http://127.0.0.1:10004/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.3455748558044434E-4,           "SEARCHER.searcher.numDocs":248},         "core_node8":{           "core":"testMixedBounds_collection_shard1_0_replica1",           "SEARCHER.searcher.maxDoc":248,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":144480,           "node_name":"127.0.0.1:10005_solr",           "base_url":"http://127.0.0.1:10005/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.3455748558044434E-4,           "SEARCHER.searcher.numDocs":248}}}}}

Stack Trace:
java.lang.AssertionError: failed to create testMixedBounds_collection
Live Nodes: [127.0.0.1:10004_solr, 127.0.0.1:10005_solr]
Last available state: DocCollection(testMixedBounds_collection//clusterstate.json/27)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testMixedBounds_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10005_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node4":{
          "core":"testMixedBounds_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10004_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1529485957954308300",
      "replicas":{
        "core_node1":{
          "core":"testMixedBounds_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10005_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495},
        "core_node2":{
          "core":"testMixedBounds_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10004_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1529485957954629200",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10005_solr",
          "base_url":"http://127.0.0.1:10005/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247},
        "core_node9":{
          "core":"testMixedBounds_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10004_solr",
          "base_url":"http://127.0.0.1:10004/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1529485957954567800",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10004_solr",
          "base_url":"http://127.0.0.1:10004/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248},
        "core_node8":{
          "core":"testMixedBounds_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10005_solr",
          "base_url":"http://127.0.0.1:10005/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248}}}}}
        at __randomizedtesting.SeedInfo.seed([A5F59192A7805BDB:AF762E3FEA3B5081]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:69)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds(IndexSizeTriggerTest.java:561)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.util.concurrent.TimeoutException: last state: DocCollection(testMixedBounds_collection//clusterstate.json/27)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testMixedBounds_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10005_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node4":{
          "core":"testMixedBounds_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10004_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1529485957954308300",
      "replicas":{
        "core_node1":{
          "core":"testMixedBounds_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10005_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495},
        "core_node2":{
          "core":"testMixedBounds_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":495,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":257740,
          "node_name":"127.0.0.1:10004_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.4003908038139343E-4,
          "SEARCHER.searcher.numDocs":495}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1529485957954629200",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10005_solr",
          "base_url":"http://127.0.0.1:10005/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247},
        "core_node9":{
          "core":"testMixedBounds_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":247,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":133740,
          "node_name":"127.0.0.1:10004_solr",
          "base_url":"http://127.0.0.1:10004/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.245550811290741E-4,
          "SEARCHER.searcher.numDocs":247}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1529485957954567800",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testMixedBounds_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10004_solr",
          "base_url":"http://127.0.0.1:10004/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248},
        "core_node8":{
          "core":"testMixedBounds_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":248,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":144480,
          "node_name":"127.0.0.1:10005_solr",
          "base_url":"http://127.0.0.1:10005/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.3455748558044434E-4,
          "SEARCHER.searcher.numDocs":248}}}}}
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:63)
        ... 39 more


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger

Error Message:
number of ops expected:<2> but was:<1>

Stack Trace:
java.lang.AssertionError: number of ops expected:<2> but was:<1>
        at __randomizedtesting.SeedInfo.seed([A5F59192A7805BDB:C63EA7103E4F28F6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger(IndexSizeTriggerTest.java:189)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:832)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: DocCollection(testSplitIntegration_collection//clusterstate.json/69)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testSplitIntegration_collection_shard2_replica_n3",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10007_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node4":{           "core":"testSplitIntegration_collection_shard2_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10006_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"0-7fffffff",       "state":"active"},     "shard1":{       "stateTimestamp":"1529486353691402950",       "replicas":{         "core_node1":{           "core":"testSplitIntegration_collection_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10007_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14},         "core_node2":{           "core":"testSplitIntegration_collection_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10006_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14}},       "range":"80000000-ffffffff",       "state":"inactive"},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1529486353691757000",       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node10":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_1_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10007_solr",           "base_url":"http://127.0.0.1:10007/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7},         "core_node9":{           "core":"testSplitIntegration_collection_shard1_1_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10006_solr",           "base_url":"http://127.0.0.1:10006/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1529486353691698500",       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node7":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_0_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10006_solr",           "base_url":"http://127.0.0.1:10006/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7},         "core_node8":{           "core":"testSplitIntegration_collection_shard1_0_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10007_solr",           "base_url":"http://127.0.0.1:10007/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7}}}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: DocCollection(testSplitIntegration_collection//clusterstate.json/69)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testSplitIntegration_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10007_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0},
        "core_node4":{
          "core":"testSplitIntegration_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10006_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1529486353691402950",
      "replicas":{
        "core_node1":{
          "core":"testSplitIntegration_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10007_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14},
        "core_node2":{
          "core":"testSplitIntegration_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10006_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1529486353691757000",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10007_solr",
          "base_url":"http://127.0.0.1:10007/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node9":{
          "core":"testSplitIntegration_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10006_solr",
          "base_url":"http://127.0.0.1:10006/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1529486353691698500",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10006_solr",
          "base_url":"http://127.0.0.1:10006/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node8":{
          "core":"testSplitIntegration_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10007_solr",
          "base_url":"http://127.0.0.1:10007/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7}}}}}
        at __randomizedtesting.SeedInfo.seed([A5F59192A7805BDB:9C7B28D2887F9225]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration(IndexSizeTriggerTest.java:301)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:832)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: DocCollection(testSplitIntegration_collection//clusterstate.json/87)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testSplitIntegration_collection_shard2_replica_n3",           "leader":"true",           "SEARCHER.searcher.maxDoc":11,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":15740,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.4659017324447632E-5,           "SEARCHER.searcher.numDocs":11},         "core_node4":{           "core":"testSplitIntegration_collection_shard2_replica_n4",           "SEARCHER.searcher.maxDoc":11,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":15740,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.4659017324447632E-5,           "SEARCHER.searcher.numDocs":11}},       "range":"0-7fffffff",       "state":"active"},     "shard1":{       "stateTimestamp":"1529481030895064250",       "replicas":{         "core_node1":{           "core":"testSplitIntegration_collection_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14},         "core_node2":{           "core":"testSplitIntegration_collection_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           "SEARCHER.searcher.numDocs":14}},       "range":"80000000-ffffffff",       "state":"inactive"},     "shard1_1":{       "parent":"shard1",       "stateTimestamp":"1529481030895541450",       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{         "core_node10":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_1_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10000_solr",           "base_url":"http://127.0.0.1:10000/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7},         "core_node9":{           "core":"testSplitIntegration_collection_shard1_1_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":13740,           "node_name":"127.0.0.1:10001_solr",           "base_url":"http://127.0.0.1:10001/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           "SEARCHER.searcher.numDocs":7}}},     "shard1_0":{       "parent":"shard1",       "stateTimestamp":"1529481030895460100",       "range":"80000000-bfffffff",       "state":"active",       "replicas":{         "core_node7":{           "leader":"true",           "core":"testSplitIntegration_collection_shard1_0_replica0",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10001_solr",           "base_url":"http://127.0.0.1:10001/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7},         "core_node8":{           "core":"testSplitIntegration_collection_shard1_0_replica1",           "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":23980,           "node_name":"127.0.0.1:10000_solr",           "base_url":"http://127.0.0.1:10000/solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           "SEARCHER.searcher.numDocs":7}}}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: DocCollection(testSplitIntegration_collection//clusterstate.json/87)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard2":{
      "replicas":{
        "core_node3":{
          "core":"testSplitIntegration_collection_shard2_replica_n3",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":11,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":15740,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.4659017324447632E-5,
          "SEARCHER.searcher.numDocs":11},
        "core_node4":{
          "core":"testSplitIntegration_collection_shard2_replica_n4",
          "SEARCHER.searcher.maxDoc":11,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":15740,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.4659017324447632E-5,
          "SEARCHER.searcher.numDocs":11}},
      "range":"0-7fffffff",
      "state":"active"},
    "shard1":{
      "stateTimestamp":"1529481030895064250",
      "replicas":{
        "core_node1":{
          "core":"testSplitIntegration_collection_shard1_replica_n1",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10001_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14},
        "core_node2":{
          "core":"testSplitIntegration_collection_shard1_replica_n2",
          "SEARCHER.searcher.maxDoc":14,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":17240,
          "node_name":"127.0.0.1:10000_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.605600118637085E-5,
          "SEARCHER.searcher.numDocs":14}},
      "range":"80000000-ffffffff",
      "state":"inactive"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1529481030895541450",
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node10":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_1_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10000_solr",
          "base_url":"http://127.0.0.1:10000/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node9":{
          "core":"testSplitIntegration_collection_shard1_1_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":13740,
          "node_name":"127.0.0.1:10001_solr",
          "base_url":"http://127.0.0.1:10001/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":1.2796372175216675E-5,
          "SEARCHER.searcher.numDocs":7}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1529481030895460100",
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "leader":"true",
          "core":"testSplitIntegration_collection_shard1_0_replica0",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10001_solr",
          "base_url":"http://127.0.0.1:10001/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7},
        "core_node8":{
          "core":"testSplitIntegration_collection_shard1_0_replica1",
          "SEARCHER.searcher.maxDoc":7,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":23980,
          "node_name":"127.0.0.1:10000_solr",
          "base_url":"http://127.0.0.1:10000/solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":2.2333115339279175E-5,
          "SEARCHER.searcher.numDocs":7}}}}}
        at __randomizedtesting.SeedInfo.seed([A5F59192A7805BDB:9C7B28D2887F9225]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration(IndexSizeTriggerTest.java:301)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:832)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger

Error Message:
number of ops expected:<2> but was:<1>

Stack Trace:
java.lang.AssertionError: number of ops expected:<2> but was:<1>
        at __randomizedtesting.SeedInfo.seed([A5F59192A7805BDB:C63EA7103E4F28F6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger(IndexSizeTriggerTest.java:189)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:832)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: DocCollection(testSplitIntegration_collection//clusterstate.json/74)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         "core_node3":{           "core":"testSplitIntegration_collection_shard2_replica_n3",           "leader":"true",           "SEARCHER.searcher.maxDoc":11,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":15740,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":

[...truncated too long message...]

cManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 116766 INFO  (zkCallback-290-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 116766 INFO  (zkCallback-277-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 116767 INFO  (jetty-closer-258-thread-1) [    ] o.a.s.c.Overseer Overseer (id=72232058244956166-127.0.0.1:36935_solr-n_0000000000) closing
   [junit4]   2> 116767 INFO  (OverseerStateUpdate-72232058244956166-127.0.0.1:36935_solr-n_0000000000) [    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:36935_solr
   [junit4]   2> 116767 WARN  (OverseerAutoScalingTriggerThread-72232058244956166-127.0.0.1:36935_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 116767 DEBUG (jetty-closer-258-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 116768 DEBUG (jetty-closer-258-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 116768 DEBUG (jetty-closer-258-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 116768 DEBUG (jetty-closer-258-thread-1) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 116768 DEBUG (jetty-closer-258-thread-1) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 116768 DEBUG (jetty-closer-258-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 116769 INFO  (zkCallback-269-thread-1) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40917_solr
   [junit4]   2> 116769 INFO  (jetty-closer-258-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7c2729fd{/solr,null,UNAVAILABLE}
   [junit4]   2> 116769 INFO  (jetty-closer-258-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 116771 INFO  (jetty-closer-258-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 116771 INFO  (jetty-closer-258-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7f320505: rootName = solr_40917, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@42a5af82
   [junit4]   2> 116772 INFO  (jetty-closer-258-thread-2) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:40917_solr
   [junit4]   2> 116773 INFO  (jetty-closer-258-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 116773 INFO  (jetty-closer-258-thread-2) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 116773 INFO  (zkCallback-290-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 116773 INFO  (zkCallback-269-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 116774 INFO  (jetty-closer-258-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@32fbb51b{/solr,null,UNAVAILABLE}
   [junit4]   2> 116774 INFO  (jetty-closer-258-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 116774 ERROR (SUITE-IndexSizeTriggerTest-seed#[A5F59192A7805BDB]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 116775 INFO  (SUITE-IndexSizeTriggerTest-seed#[A5F59192A7805BDB]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42753 42753
   [junit4]   2> 117192 INFO  (Thread-1620) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42753 42753
   [junit4]   2> 117192 WARN  (Thread-1620) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/aliases.json
   [junit4]   2> 2 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/clusterstate.json
   [junit4]   2> 5 /solr/clusterprops.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 5 /solr/live_nodes
   [junit4]   2> 5 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.IndexSizeTriggerTest_A5F59192A7805BDB-004
   [junit4]   2> Jun 20, 2018 7:50:04 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=2004, maxMBSortInHeap=6.7462155014282565, sim=RandomSimilarity(queryNorm=false): {}, locale=en-IN, timezone=UTC
   [junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 11-ea (64-bit)/cpus=8,threads=1,free=177547288,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [IndexSizeTriggerTest, MoveReplicaHDFSTest, IndexSizeTriggerTest, IndexSizeTriggerTest, IndexSizeTriggerTest]
   [junit4] Completed [7/10 (4!)] on J2 in 3.49s, 4 tests, 1 error <<< FAILURES!
   [junit4]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
   [junit4] OK      0.78s J2 | IndexSizeTriggerTest.testMixedBounds
   [junit4] OK      0.67s J2 | IndexSizeTriggerTest.testSplitIntegration
   [junit4] OK      0.30s J2 | IndexSizeTriggerTest.testTrigger
   [junit4] OK      0.34s J2 | IndexSizeTriggerTest.testMergeIntegration
   [junit4] Completed [8/10 (4!)] on J2 in 5.14s, 4 tests
   [junit4]
   [junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
   [junit4] OK      18.3s J1 | MoveReplicaHDFSTest.testNormalMove
   [junit4] OK      20.6s J1 | MoveReplicaHDFSTest.test
   [junit4] OK      13.2s J1 | MoveReplicaHDFSTest.testFailedMove
   [junit4] OK      12.7s J1 | MoveReplicaHDFSTest.testNormalFailedMove
   [junit4] Completed [9/10 (4!)] on J1 in 81.19s, 4 tests
   [junit4]
   [junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
   [junit4] OK      19.2s J0 | MoveReplicaHDFSTest.testNormalMove
   [junit4] OK      20.6s J0 | MoveReplicaHDFSTest.test
   [junit4] OK      15.7s J0 | MoveReplicaHDFSTest.testFailedMove
   [junit4] OK      12.5s J0 | MoveReplicaHDFSTest.testNormalFailedMove
   [junit4] Completed [10/10 (4!)] on J0 in 80.96s, 4 tests
   [junit4]
   [junit4]
   [junit4] Tests with failures [seed: A5F59192A7805BDB]:
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger
   [junit4]   - org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration
   [junit4]
   [junit4]
   [junit4] JVM J0:     0.80 ..   183.43 =   182.63s
   [junit4] JVM J1:     0.79 ..   177.41 =   176.62s
   [junit4] JVM J2:     0.82 ..   123.32 =   122.49s
   [junit4] Execution time total: 3 minutes 3 seconds
   [junit4] Tests summary: 10 suites, 40 tests, 3 errors, 4 failures

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/common-build.xml:1568: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/common-build.xml:1092: There were test failures: 10 suites, 40 tests, 3 errors, 4 failures [seed: A5F59192A7805BDB]

Total time: 3 minutes 5 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.MoveReplicaHDFSTest
[repro]   4/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=9efe619afa636d18362f38ba651bf8bc7244c6a9, workspace=/var/lib/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux
[WARNINGS] Computing warning deltas based on reference build #52
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]