[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4919 - Unstable!

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4919 - Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4919/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

6 tests failed.
FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:58931/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:58931/collection1
        at __randomizedtesting.SeedInfo.seed([3B63CE704A0CB3F4:B337F1AAE4F0DE0C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:59952/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:59952/collection1
        at __randomizedtesting.SeedInfo.seed([3B63CE704A0CB3F4:B337F1AAE4F0DE0C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:59965/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:59965/collection1
        at __randomizedtesting.SeedInfo.seed([3B63CE704A0CB3F4:B337F1AAE4F0DE0C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:60964/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:60964/collection1
        at __randomizedtesting.SeedInfo.seed([3B63CE704A0CB3F4:B337F1AAE4F0DE0C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:58934/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:58934/collection1
        at __randomizedtesting.SeedInfo.seed([3B63CE704A0CB3F4:B337F1AAE4F0DE0C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:50230/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:50230/collection1
        at __randomizedtesting.SeedInfo.seed([3B63CE704A0CB3F4:B337F1AAE4F0DE0C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more




Build Log:
[...truncated 14194 lines...]
   [junit4] Suite: org.apache.solr.cloud.RecoveryAfterSoftCommitTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/init-core-data-001
   [junit4]   2> 2382211 INFO  (SUITE-RecoveryAfterSoftCommitTest-seed#[3B63CE704A0CB3F4]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2382213 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2382213 INFO  (Thread-4539) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2382213 INFO  (Thread-4539) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2382216 ERROR (Thread-4539) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2382319 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.ZkTestServer start zk server on port:50181
   [junit4]   2> 2382329 INFO  (zkConnectionManagerCallback-9681-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2382333 INFO  (zkConnectionManagerCallback-9683-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2382346 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2382414 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2382420 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2382423 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2382428 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2382431 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2382443 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2382502 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2382533 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2382630 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2382687 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2382739 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 2384902 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2384903 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2384903 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2384903 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2384904 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3131a29a{/,null,AVAILABLE}
   [junit4]   2> 2384904 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@769f6e3a{HTTP/1.1,[http/1.1]}{127.0.0.1:50206}
   [junit4]   2> 2384904 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.Server Started @2385012ms
   [junit4]   2> 2384904 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=50205, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/control-001/cores}
   [junit4]   2> 2384905 ERROR (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2384905 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2384905 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 2384905 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2384905 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2384905 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-08T18:09:47.309Z
   [junit4]   2> 2384907 INFO  (zkConnectionManagerCallback-9685-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2384908 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2384908 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/control-001/solr.xml
   [junit4]   2> 2384915 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2384915 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2384916 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@612b2f4, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2385196 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50181/solr
   [junit4]   2> 2385197 INFO  (zkConnectionManagerCallback-9689-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2385203 INFO  (zkConnectionManagerCallback-9691-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2385389 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [n:127.0.0.1:50205_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50205_
   [junit4]   2> 2385391 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [n:127.0.0.1:50205_    ] o.a.s.c.Overseer Overseer (id=72077083911585796-127.0.0.1:50205_-n_0000000000) starting
   [junit4]   2> 2385403 INFO  (zkConnectionManagerCallback-9698-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2385407 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [n:127.0.0.1:50205_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50181/solr ready
   [junit4]   2> 2385408 INFO  (OverseerStateUpdate-72077083911585796-127.0.0.1:50205_-n_0000000000) [n:127.0.0.1:50205_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:50205_
   [junit4]   2> 2385410 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [n:127.0.0.1:50205_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50205_
   [junit4]   2> 2385412 INFO  (zkCallback-9690-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2385412 INFO  (zkCallback-9697-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2385458 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [n:127.0.0.1:50205_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2385481 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [n:127.0.0.1:50205_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@612b2f4
   [junit4]   2> 2385488 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [n:127.0.0.1:50205_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@612b2f4
   [junit4]   2> 2385488 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [n:127.0.0.1:50205_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@612b2f4
   [junit4]   2> 2385489 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [n:127.0.0.1:50205_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/control-001/cores
   [junit4]   2> 2385517 INFO  (zkConnectionManagerCallback-9704-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2385518 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2385520 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50181/solr ready
   [junit4]   2> 2385521 INFO  (SocketProxy-Acceptor-50205) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50212,localport=50205], receiveBufferSize:408300
   [junit4]   2> 2385521 INFO  (SocketProxy-Acceptor-50205) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=50206,localport=50213], receiveBufferSize=408300
   [junit4]   2> 2385522 INFO  (qtp908137059-26991) [n:127.0.0.1:50205_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:50205_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2385526 INFO  (OverseerThreadFactory-9909-thread-1-processing-n:127.0.0.1:50205_) [n:127.0.0.1:50205_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2385531 DEBUG (OverseerStateUpdate-72077083911585796-127.0.0.1:50205_-n_0000000000) [n:127.0.0.1:50205_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"control_collection",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "createNodeSet":"127.0.0.1:50205_",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 2385640 INFO  (SocketProxy-Acceptor-50205) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50214,localport=50205], receiveBufferSize:408300
   [junit4]   2> 2385640 INFO  (SocketProxy-Acceptor-50205) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=50206,localport=50215], receiveBufferSize=408300
   [junit4]   2> 2385642 INFO  (qtp908137059-26987) [n:127.0.0.1:50205_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2385647 INFO  (SocketProxy-Acceptor-50205) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50216,localport=50205], receiveBufferSize:408300
   [junit4]   2> 2385648 INFO  (SocketProxy-Acceptor-50205) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=50206,localport=50217], receiveBufferSize=408300
   [junit4]   2> 2385649 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2385650 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2385656 DEBUG (OverseerStateUpdate-72077083911585796-127.0.0.1:50205_-n_0000000000) [n:127.0.0.1:50205_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:50205",
   [junit4]   2>   "node_name":"127.0.0.1:50205_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2386685 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2386710 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2386800 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2386826 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2386826 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@612b2f4
   [junit4]   2> 2386826 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2386827 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3533ee5b
   [junit4]   2> 2386830 WARN  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2386884 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2386884 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=2 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2386885 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2386885 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2386886 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7500097278573631]
   [junit4]   2> 2386886 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@17ee79a8[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2386888 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2386890 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2386890 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2386890 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1616590237119545344
   [junit4]   2> 2386891 INFO  (searcherExecutor-9914-thread-1-processing-n:127.0.0.1:50205_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@17ee79a8[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2386898 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 2386904 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2386904 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2386904 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:50205/control_collection_shard1_replica_n1/
   [junit4]   2> 2386905 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2386905 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:50205/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2386912 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:50205/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2386913 DEBUG (OverseerStateUpdate-72077083911585796-127.0.0.1:50205_-n_0000000000) [n:127.0.0.1:50205_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:50205",
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "state":"active"} current state version: 0
   [junit4]   2> 2387080 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2387086 DEBUG (OverseerStateUpdate-72077083911585796-127.0.0.1:50205_-n_0000000000) [n:127.0.0.1:50205_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "core_node_name":"core_node2",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:50205",
   [junit4]   2>   "node_name":"127.0.0.1:50205_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2387087 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1438
   [junit4]   2> 2387090 INFO  (qtp908137059-26991) [n:127.0.0.1:50205_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2387197 INFO  (zkCallback-9690-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2387539 INFO  (OverseerCollectionConfigSetProcessor-72077083911585796-127.0.0.1:50205_-n_0000000000) [n:127.0.0.1:50205_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2388090 INFO  (qtp908137059-26991) [n:127.0.0.1:50205_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:50205_&wt=javabin&version=2} status=0 QTime=2568
   [junit4]   2> 2388097 INFO  (zkConnectionManagerCallback-9709-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2388098 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2388100 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50181/solr ready
   [junit4]   2> 2388100 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2388101 INFO  (SocketProxy-Acceptor-50205) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50222,localport=50205], receiveBufferSize:408300
   [junit4]   2> 2388101 INFO  (SocketProxy-Acceptor-50205) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=50206,localport=50223], receiveBufferSize=408300
   [junit4]   2> 2388102 INFO  (qtp908137059-26993) [n:127.0.0.1:50205_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2388106 INFO  (OverseerThreadFactory-9909-thread-2-processing-n:127.0.0.1:50205_) [n:127.0.0.1:50205_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2388111 DEBUG (OverseerStateUpdate-72077083911585796-127.0.0.1:50205_-n_0000000000) [n:127.0.0.1:50205_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "createNodeSet":"",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 2388316 WARN  (OverseerThreadFactory-9909-thread-2-processing-n:127.0.0.1:50205_) [n:127.0.0.1:50205_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2388319 INFO  (qtp908137059-26993) [n:127.0.0.1:50205_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2388319 INFO  (qtp908137059-26993) [n:127.0.0.1:50205_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=217
   [junit4]   2> 2388704 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/shard-1-001 of type TLOG
   [junit4]   2> 2388705 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2388706 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2388706 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2388706 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2388707 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@67e25d6a{/,null,AVAILABLE}
   [junit4]   2> 2388707 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2e2a24c0{HTTP/1.1,[http/1.1]}{127.0.0.1:50231}
   [junit4]   2> 2388707 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.e.j.s.Server Started @2388815ms
   [junit4]   2> 2388707 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=50230, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/shard-1-001/cores}
   [junit4]   2> 2388707 ERROR (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2388707 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2388707 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 2388708 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2388708 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2388708 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-08T18:09:51.112Z
   [junit4]   2> 2388709 INFO  (zkConnectionManagerCallback-9711-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2388710 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2388711 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/shard-1-001/solr.xml
   [junit4]   2> 2388726 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2388726 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2388727 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@612b2f4, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2388963 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50181/solr
   [junit4]   2> 2388964 INFO  (zkConnectionManagerCallback-9715-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2388970 INFO  (zkConnectionManagerCallback-9717-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2388982 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2388990 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2388990 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50230_
   [junit4]   2> 2388990 DEBUG (OverseerStateUpdate-72077083911585796-127.0.0.1:50205_-n_0000000000) [n:127.0.0.1:50205_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:50230_"} current state version: 0
   [junit4]   2> 2388991 INFO  (zkCallback-9690-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2388991 INFO  (zkCallback-9708-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2388991 INFO  (zkCallback-9697-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2388992 INFO  (zkCallback-9716-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2389031 INFO  (zkConnectionManagerCallback-9724-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2389033 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2389035 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50181/solr ready
   [junit4]   2> 2389035 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2389058 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@612b2f4
   [junit4]   2> 2389065 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@612b2f4
   [junit4]   2> 2389065 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@612b2f4
   [junit4]   2> 2389066 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/shard-1-001/cores
   [junit4]   2> 2389095 INFO  (qtp908137059-26990) [n:127.0.0.1:50205_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:50230_&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2389100 INFO  (OverseerCollectionConfigSetProcessor-72077083911585796-127.0.0.1:50205_-n_0000000000) [n:127.0.0.1:50205_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2389105 INFO  (qtp908137059-26993) [n:127.0.0.1:50205_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 2389106 INFO  (qtp908137059-26989) [n:127.0.0.1:50205_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2389107 INFO  (SocketProxy-Acceptor-50230) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50238,localport=50230], receiveBufferSize:408300
   [junit4]   2> 2389108 INFO  (SocketProxy-Acceptor-50230) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=50231,localport=50239], receiveBufferSize=408300
   [junit4]   2> 2389109 INFO  (qtp1633157564-27054) [n:127.0.0.1:50230_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2389110 INFO  (OverseerThreadFactory-9909-thread-3-processing-n:127.0.0.1:50205_) [n:127.0.0.1:50205_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:50230_ for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 2389114 INFO  (SocketProxy-Acceptor-50230) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50240,localport=50230], receiveBufferSize:408300
   [junit4]   2> 2389114 INFO  (SocketProxy-Acceptor-50230) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=50231,localport=50241], receiveBufferSize=408300
   [junit4]   2> 2389116 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_    x:collection1_shard1_replica_t21] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2389120 DEBUG (OverseerStateUpdate-72077083911585796-127.0.0.1:50205_-n_0000000000) [n:127.0.0.1:50205_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1_shard1_replica_t21",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:50230",
   [junit4]   2>   "node_name":"127.0.0.1:50230_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2390144 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2390155 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.IndexSchema [collection1_shard1_replica_t21] Schema name=test
   [junit4]   2> 2390382 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2390407 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t21' using configuration from collection collection1, trusted=true
   [junit4]   2> 2390407 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t21' (registry 'solr.core.collection1.shard1.replica_t21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@612b2f4
   [junit4]   2> 2390407 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrCore [[collection1_shard1_replica_t21] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/shard-1-001/cores/collection1_shard1_replica_t21], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-001/shard-1-001/cores/collection1_shard1_replica_t21/data/]
   [junit4]   2> 2390409 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@14fd2cef
   [junit4]   2> 2390412 WARN  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2390590 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2390590 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=2 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2390591 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2390591 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2390595 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7500097278573631]
   [junit4]   2> 2390622 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.SolrIndexSearcher Opening [Searcher@2321af21[collection1_shard1_replica_t21] main]
   [junit4]   2> 2390623 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2390625 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2390627 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2390627 INFO  (qtp1633157564-27050) [n:127.0.0.1:50230_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1616590241038073856
   [junit4]   2> 2390627 INFO  (searcherExecutor-9928-thread-1-processing-n:127.0.0.1:50230_ x:collection1_shard1_replica_t21 c:collection1 s:sh

[...truncated too long message...]

veryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60945 60945
   [junit4]   2> 249422 INFO  (Thread-58) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60945 60945
   [junit4]   2> 249548 WARN  (Thread-58) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/aliases.json
   [junit4]   2> 7 /solr/collections/collection1/terms/shard1
   [junit4]   2> 3 /solr/security.json
   [junit4]   2> 3 /solr/configs/conf1
   [junit4]   2> 2 /solr/collections/control_collection/terms/shard1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/clusterstate.json
   [junit4]   2> 8 /solr/clusterprops.json
   [junit4]   2> 7 /solr/collections/collection1/state.json
   [junit4]   2> 2 /solr/autoscaling.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/live_nodes
   [junit4]   2> 8 /solr/collections
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> 249548 WARN  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SocketProxy Closing 4 connections to: http://127.0.0.1:60948/, target: http://127.0.0.1:60949/
   [junit4]   2> 249549 WARN  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SocketProxy Closing 239 connections to: http://127.0.0.1:60964/, target: http://127.0.0.1:60965/
   [junit4]   2> 249549 WARN  (TEST-RecoveryAfterSoftCommitTest.test-seed#[3B63CE704A0CB3F4]) [    ] o.a.s.c.SocketProxy Closing 3 connections to: http://127.0.0.1:60974/, target: http://127.0.0.1:60975/
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RecoveryAfterSoftCommitTest -Dtests.method=test -Dtests.seed=3B63CE704A0CB3F4 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=he-IL -Dtests.timezone=Canada/Mountain -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   73.2s J0 | RecoveryAfterSoftCommitTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:60964/collection1
   [junit4]    > at __randomizedtesting.SeedInfo.seed([3B63CE704A0CB3F4:B337F1AAE4F0DE0C]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
   [junit4]    > at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]    > at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]    > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]    > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]    > at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > ... 50 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_3B63CE704A0CB3F4-003
   [junit4]   2> Nov 08, 2018 6:59:15 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@23f8f748), locale=he-IL, timezone=Canada/Mountain
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=3,threads=1,free=57752608,total=261619712
   [junit4]   2> NOTE: All tests run in this JVM: [RecoveryAfterSoftCommitTest, RecoveryAfterSoftCommitTest, RecoveryAfterSoftCommitTest]
   [junit4] Completed [5/5 (5!)] on J0 in 73.47s, 1 test, 1 error <<< FAILURES!
   [junit4]
   [junit4]
   [junit4] Tests with failures [seed: 3B63CE704A0CB3F4]:
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]
   [junit4]
   [junit4] JVM J0:     0.70 ..   250.69 =   249.99s
   [junit4] JVM J1:     0.70 ..   177.60 =   176.90s
   [junit4] Execution time total: 4 minutes 10 seconds
   [junit4] Tests summary: 5 suites, 5 tests, 5 errors

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1567: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1092: There were test failures: 5 suites, 5 tests, 5 errors [seed: 3B63CE704A0CB3F4]

Total time: 4 minutes 12 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   5/5 failed: org.apache.solr.cloud.RecoveryAfterSoftCommitTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=1b084db9017f108115a3081726359695bd5a8ff1, workspace=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX
[WARNINGS] Computing warning deltas based on reference build #4918
Recording test results
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4920 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4920/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

8 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Expected new active leader null Live Nodes: [127.0.0.1:49779_solr, 127.0.0.1:49780_solr, 127.0.0.1:49781_solr] Last available state: DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/13)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"raceDeleteReplica_false_shard1_replica_n1",           "base_url":"http://127.0.0.1:49782/solr",           "node_name":"127.0.0.1:49782_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node6":{           "core":"raceDeleteReplica_false_shard1_replica_n5",           "base_url":"http://127.0.0.1:49782/solr",           "node_name":"127.0.0.1:49782_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected new active leader
null
Live Nodes: [127.0.0.1:49779_solr, 127.0.0.1:49780_solr, 127.0.0.1:49781_solr]
Last available state: DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/13)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"raceDeleteReplica_false_shard1_replica_n1",
          "base_url":"http://127.0.0.1:49782/solr",
          "node_name":"127.0.0.1:49782_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node6":{
          "core":"raceDeleteReplica_false_shard1_replica_n5",
          "base_url":"http://127.0.0.1:49782/solr",
          "node_name":"127.0.0.1:49782_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([6583D3DC3D2671FC:F95B20C55D43B36]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:328)
        at org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:224)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards

Error Message:
Error from server at http://127.0.0.1:49779/solr: Cannot create collection deleteByCountNew. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:49779/solr: Cannot create collection deleteByCountNew. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)
        at __randomizedtesting.SeedInfo.seed([6583D3DC3D2671FC:5912948872672CA9]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards(DeleteReplicaTest.java:161)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest

Error Message:
Error from server at http://127.0.0.1:49779/solr: Cannot create collection delLiveColl. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:49779/solr: Cannot create collection delLiveColl. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)
        at __randomizedtesting.SeedInfo.seed([6583D3DC3D2671FC:C8E367D72019D989]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:73)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Expected new active leader null Live Nodes: [127.0.0.1:50655_solr, 127.0.0.1:50656_solr, 127.0.0.1:50659_solr] Last available state: DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/13)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"raceDeleteReplica_false_shard1_replica_n1",           "base_url":"http://127.0.0.1:50660/solr",           "node_name":"127.0.0.1:50660_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node6":{           "core":"raceDeleteReplica_false_shard1_replica_n5",           "base_url":"http://127.0.0.1:50660/solr",           "node_name":"127.0.0.1:50660_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected new active leader
null
Live Nodes: [127.0.0.1:50655_solr, 127.0.0.1:50656_solr, 127.0.0.1:50659_solr]
Last available state: DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/13)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"raceDeleteReplica_false_shard1_replica_n1",
          "base_url":"http://127.0.0.1:50660/solr",
          "node_name":"127.0.0.1:50660_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node6":{
          "core":"raceDeleteReplica_false_shard1_replica_n5",
          "base_url":"http://127.0.0.1:50660/solr",
          "node_name":"127.0.0.1:50660_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([6583D3DC3D2671FC:F95B20C55D43B36]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:328)
        at org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:224)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards

Error Message:
Error from server at http://127.0.0.1:50655/solr: Cannot create collection deleteByCountNew. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:50655/solr: Cannot create collection deleteByCountNew. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)
        at __randomizedtesting.SeedInfo.seed([6583D3DC3D2671FC:5912948872672CA9]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards(DeleteReplicaTest.java:161)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Expected new active leader null Live Nodes: [127.0.0.1:54681_solr, 127.0.0.1:54682_solr, 127.0.0.1:54683_solr] Last available state: DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/13)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node4":{           "core":"raceDeleteReplica_false_shard1_replica_n2",           "base_url":"http://127.0.0.1:54684/solr",           "node_name":"127.0.0.1:54684_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node6":{           "core":"raceDeleteReplica_false_shard1_replica_n5",           "base_url":"http://127.0.0.1:54684/solr",           "node_name":"127.0.0.1:54684_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected new active leader
null
Live Nodes: [127.0.0.1:54681_solr, 127.0.0.1:54682_solr, 127.0.0.1:54683_solr]
Last available state: DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/13)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"raceDeleteReplica_false_shard1_replica_n2",
          "base_url":"http://127.0.0.1:54684/solr",
          "node_name":"127.0.0.1:54684_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node6":{
          "core":"raceDeleteReplica_false_shard1_replica_n5",
          "base_url":"http://127.0.0.1:54684/solr",
          "node_name":"127.0.0.1:54684_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([6583D3DC3D2671FC:F95B20C55D43B36]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:328)
        at org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:224)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards

Error Message:
Error from server at http://127.0.0.1:54682/solr: Cannot create collection deleteByCountNew. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:54682/solr: Cannot create collection deleteByCountNew. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)
        at __randomizedtesting.SeedInfo.seed([6583D3DC3D2671FC:5912948872672CA9]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards(DeleteReplicaTest.java:161)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest

Error Message:
Error from server at http://127.0.0.1:54682/solr: Cannot create collection delLiveColl. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:54682/solr: Cannot create collection delLiveColl. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 3. This allows a maximum of 3 to be created. Value of numShards is 2, value of nrtReplicas is 2, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 4 shards to be created (higher than the allowed number)
        at __randomizedtesting.SeedInfo.seed([6583D3DC3D2671FC:C8E367D72019D989]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:73)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14146 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-001/init-core-data-001
   [junit4]   2> 2003125 WARN  (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 2003125 INFO  (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2003129 INFO  (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2003129 INFO  (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2003130 INFO  (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-001/tempDir-001
   [junit4]   2> 2003130 INFO  (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2003130 INFO  (Thread-3822) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2003130 INFO  (Thread-3822) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2003133 ERROR (Thread-3822) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2003240 INFO  (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:54679
   [junit4]   2> 2003253 INFO  (zkConnectionManagerCallback-6074-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003257 INFO  (jetty-launcher-6071-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2003258 INFO  (jetty-launcher-6071-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2003258 INFO  (jetty-launcher-6071-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2003258 INFO  (jetty-launcher-6071-thread-4) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2003259 INFO  (jetty-launcher-6071-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2003259 INFO  (jetty-launcher-6071-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2003259 INFO  (jetty-launcher-6071-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2003260 INFO  (jetty-launcher-6071-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2003260 INFO  (jetty-launcher-6071-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2003260 INFO  (jetty-launcher-6071-thread-3) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2003260 INFO  (jetty-launcher-6071-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4461278b{/solr,null,AVAILABLE}
   [junit4]   2> 2003261 INFO  (jetty-launcher-6071-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f7a61bb{/solr,null,AVAILABLE}
   [junit4]   2> 2003261 INFO  (jetty-launcher-6071-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2003261 INFO  (jetty-launcher-6071-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2003261 INFO  (jetty-launcher-6071-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2003262 INFO  (jetty-launcher-6071-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@69bd79fc{HTTP/1.1,[http/1.1]}{127.0.0.1:54681}
   [junit4]   2> 2003262 INFO  (jetty-launcher-6071-thread-1) [    ] o.e.j.s.Server Started @2003338ms
   [junit4]   2> 2003262 INFO  (jetty-launcher-6071-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2003262 INFO  (jetty-launcher-6071-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54681}
   [junit4]   2> 2003262 INFO  (jetty-launcher-6071-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2003262 INFO  (jetty-launcher-6071-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3cf3522d{HTTP/1.1,[http/1.1]}{127.0.0.1:54682}
   [junit4]   2> 2003262 INFO  (jetty-launcher-6071-thread-4) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2003262 INFO  (jetty-launcher-6071-thread-3) [    ] o.e.j.s.Server Started @2003338ms
   [junit4]   2> 2003262 INFO  (jetty-launcher-6071-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54682}
   [junit4]   2> 2003263 ERROR (jetty-launcher-6071-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2003263 ERROR (jetty-launcher-6071-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-09T05:51:19.312Z
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7046b0bc{/solr,null,AVAILABLE}
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-09T05:51:19.312Z
   [junit4]   2> 2003263 INFO  (jetty-launcher-6071-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6c91bd8a{/solr,null,AVAILABLE}
   [junit4]   2> 2003264 INFO  (jetty-launcher-6071-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2670db01{HTTP/1.1,[http/1.1]}{127.0.0.1:54683}
   [junit4]   2> 2003264 INFO  (jetty-launcher-6071-thread-4) [    ] o.e.j.s.Server Started @2003340ms
   [junit4]   2> 2003264 INFO  (jetty-launcher-6071-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54683}
   [junit4]   2> 2003264 ERROR (jetty-launcher-6071-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2003264 INFO  (jetty-launcher-6071-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2003264 INFO  (jetty-launcher-6071-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2003264 INFO  (jetty-launcher-6071-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2003264 INFO  (jetty-launcher-6071-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2003264 INFO  (jetty-launcher-6071-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-09T05:51:19.313Z
   [junit4]   2> 2003265 INFO  (jetty-launcher-6071-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@25aa2920{HTTP/1.1,[http/1.1]}{127.0.0.1:54684}
   [junit4]   2> 2003265 INFO  (jetty-launcher-6071-thread-2) [    ] o.e.j.s.Server Started @2003341ms
   [junit4]   2> 2003265 INFO  (jetty-launcher-6071-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54684}
   [junit4]   2> 2003265 ERROR (jetty-launcher-6071-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2003265 INFO  (jetty-launcher-6071-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2003265 INFO  (jetty-launcher-6071-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2003265 INFO  (jetty-launcher-6071-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2003265 INFO  (jetty-launcher-6071-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2003265 INFO  (jetty-launcher-6071-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-09T05:51:19.314Z
   [junit4]   2> 2003267 INFO  (zkConnectionManagerCallback-6076-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003267 INFO  (zkConnectionManagerCallback-6078-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003268 INFO  (zkConnectionManagerCallback-6080-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003268 INFO  (zkConnectionManagerCallback-6082-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003268 INFO  (jetty-launcher-6071-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2003268 INFO  (jetty-launcher-6071-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2003268 INFO  (jetty-launcher-6071-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2003269 INFO  (jetty-launcher-6071-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2003339 INFO  (jetty-launcher-6071-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54679/solr
   [junit4]   2> 2003360 INFO  (zkConnectionManagerCallback-6086-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003394 INFO  (zkConnectionManagerCallback-6088-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003522 INFO  (jetty-launcher-6071-thread-3) [n:127.0.0.1:54682_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54682_solr
   [junit4]   2> 2003523 INFO  (jetty-launcher-6071-thread-3) [n:127.0.0.1:54682_solr    ] o.a.s.c.Overseer Overseer (id=72079842620866566-127.0.0.1:54682_solr-n_0000000000) starting
   [junit4]   2> 2003530 INFO  (zkConnectionManagerCallback-6095-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003533 INFO  (jetty-launcher-6071-thread-3) [n:127.0.0.1:54682_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54679/solr ready
   [junit4]   2> 2003535 INFO  (OverseerStateUpdate-72079842620866566-127.0.0.1:54682_solr-n_0000000000) [n:127.0.0.1:54682_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:54682_solr
   [junit4]   2> 2003535 INFO  (jetty-launcher-6071-thread-3) [n:127.0.0.1:54682_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54682_solr
   [junit4]   2> 2003537 INFO  (zkCallback-6094-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2003538 INFO  (zkCallback-6087-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2003550 INFO  (jetty-launcher-6071-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54679/solr
   [junit4]   2> 2003552 INFO  (jetty-launcher-6071-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54679/solr
   [junit4]   2> 2003552 INFO  (zkConnectionManagerCallback-6100-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003553 INFO  (zkConnectionManagerCallback-6103-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003555 INFO  (zkConnectionManagerCallback-6105-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003556 INFO  (zkConnectionManagerCallback-6107-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003573 INFO  (jetty-launcher-6071-thread-1) [n:127.0.0.1:54681_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2003573 INFO  (jetty-launcher-6071-thread-2) [n:127.0.0.1:54684_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2003582 INFO  (jetty-launcher-6071-thread-3) [n:127.0.0.1:54682_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2003583 INFO  (jetty-launcher-6071-thread-2) [n:127.0.0.1:54684_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2003583 INFO  (jetty-launcher-6071-thread-2) [n:127.0.0.1:54684_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54684_solr
   [junit4]   2> 2003584 INFO  (jetty-launcher-6071-thread-1) [n:127.0.0.1:54681_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2003584 INFO  (jetty-launcher-6071-thread-1) [n:127.0.0.1:54681_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54681_solr
   [junit4]   2> 2003585 INFO  (zkCallback-6094-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2003585 INFO  (zkCallback-6106-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2003585 INFO  (zkCallback-6087-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2003586 INFO  (zkCallback-6104-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2003587 INFO  (zkCallback-6104-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2003587 INFO  (zkCallback-6094-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2003587 INFO  (zkCallback-6106-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2003587 INFO  (zkCallback-6087-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2003609 INFO  (zkConnectionManagerCallback-6117-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003610 INFO  (jetty-launcher-6071-thread-2) [n:127.0.0.1:54684_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2003611 INFO  (jetty-launcher-6071-thread-2) [n:127.0.0.1:54684_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54679/solr ready
   [junit4]   2> 2003612 INFO  (jetty-launcher-6071-thread-2) [n:127.0.0.1:54684_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2003621 INFO  (zkConnectionManagerCallback-6123-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003624 INFO  (jetty-launcher-6071-thread-1) [n:127.0.0.1:54681_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2003626 INFO  (jetty-launcher-6071-thread-1) [n:127.0.0.1:54681_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54679/solr ready
   [junit4]   2> 2003627 INFO  (jetty-launcher-6071-thread-1) [n:127.0.0.1:54681_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2003647 INFO  (jetty-launcher-6071-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54679/solr
   [junit4]   2> 2003649 INFO  (zkConnectionManagerCallback-6128-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003651 INFO  (zkConnectionManagerCallback-6130-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2003663 INFO  (jetty-launcher-6071-thread-4) [n:127.0.0.1:54683_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2003668 INFO  (jetty-launcher-6071-thread-3) [n:127.0.0.1:54682_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54682.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2003668 INFO  (jetty-launcher-6071-thread-1) [n:127.0.0.1:54681_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54681.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2003674 INFO  (jetty-launcher-6071-thread-4) [n:127.0.0.1:54683_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2003674 INFO  (jetty-launcher-6071-thread-4) [n:127.0.0.1:54683_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54683_solr
   [junit4]   2> 2003677 INFO  (zkCallback-6129-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2003677 INFO  (zkCallback-6094-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2003677 INFO  (zkCallback-6122-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2003677 INFO  (zkCallback-6116-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2003677 INFO  (zkCallback-6106-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2003677 INFO  (zkCallback-6104-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2003678 INFO  (zkCallback-6087-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2004433 INFO  (jetty-launcher-6071-thread-3) [n:127.0.0.1:54682_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54682.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2004433 INFO  (jetty-launcher-6071-thread-3) [n:127.0.0.1:54682_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54682.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2004437 INFO  (jetty-launcher-6071-thread-3) [n:127.0.0.1:54682_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-001/tempDir-001/node3/.
   [junit4]   2> 2004440 INFO  (jetty-launcher-6071-thread-2) [n:127.0.0.1:54684_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54684.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2004440 INFO  (jetty-launcher-6071-thread-1) [n:127.0.0.1:54681_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54681.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2004440 INFO  (jetty-launcher-6071-thread-1) [n:127.0.0.1:54681_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54681.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2004441 INFO  (jetty-launcher-6071-thread-1) [n:127.0.0.1:54681_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-001/tempDir-001/node1/.
   [junit4]   2> 2004445 INFO  (jetty-launcher-6071-thread-2) [n:127.0.0.1:54684_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54684.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2004445 INFO  (jetty-launcher-6071-thread-2) [n:127.0.0.1:54684_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54684.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2004446 INFO  (jetty-launcher-6071-thread-2) [n:127.0.0.1:54684_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-001/tempDir-001/node2/.
   [junit4]   2> 2004454 INFO  (zkConnectionManagerCallback-6137-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2004456 INFO  (jetty-launcher-6071-thread-4) [n:127.0.0.1:54683_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2004458 INFO  (jetty-launcher-6071-thread-4) [n:127.0.0.1:54683_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54679/solr ready
   [junit4]   2> 2004458 INFO  (jetty-launcher-6071-thread-4) [n:127.0.0.1:54683_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2004489 INFO  (jetty-launcher-6071-thread-4) [n:127.0.0.1:54683_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54683.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2004496 INFO  (jetty-launcher-6071-thread-4) [n:127.0.0.1:54683_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54683.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2004496 INFO  (jetty-launcher-6071-thread-4) [n:127.0.0.1:54683_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54683.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2004497 INFO  (jetty-launcher-6071-thread-4) [n:127.0.0.1:54683_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-001/tempDir-001/node4/.
   [junit4]   2> 2004550 INFO  (zkConnectionManagerCallback-6140-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2004556 INFO  (zkConnectionManagerCallback-6145-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2004558 INFO  (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2004560 INFO  (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54679/solr ready
   [junit4]   2> 2004594 INFO  (TEST-DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup-seed#[6583D3DC3D2671FC]) [    ] o.a.s.SolrTestCaseJ4 ###Starting deleteReplicaAndVerifyDirectoryCleanup
   [junit4]   2> 2004596 INFO  (qtp1593290629-21160) [n:127.0.0.1:54684_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&name=deletereplica_test&nrtReplicas=2&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2004600 INFO  (OverseerThreadFactory-10132-thread-1-processing-n:127.0.0.1:54682_solr) [n:127.0.0.1:54682_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection deletereplica_test
   [junit4]   2> 2004713 INFO  (qtp1593290629-21151) [n:127.0.0.1:54684_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2004717 INFO  (qtp1045280163-21141) [n:127.0.0.1:54681_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2004719 INFO  (qtp711349975-21168) [n:127.0.0.1:54683_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2004721 INFO  (qtp17530944-21149) [n:127.0.0.1:54682_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2004728 INFO  (OverseerStateUpdate-72079842620866566-127.0.0.1:54682_solr-n_0000000000) [n:127.0.0.1:54682_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deletereplica_test",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deletereplica_test_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:54682/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2004733 INFO  (OverseerStateUpdate-72079842620866566-127.0.0.1:54682_solr-n_0000000000) [n:127.0.0.1:54682_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deletereplica_test",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deletereplica_test_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:54683/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2004938 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr    x:deletereplica_test_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deletereplica_test_shard1_replica_n1&action=CREATE&numShards=1&collection=deletereplica_test&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2004938 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr    x:deletereplica_test_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=deletereplica_test_shard1_replica_n2&action=CREATE&numShards=1&collection=deletereplica_test&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2004939 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr    x:deletereplica_test_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2005965 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2005978 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2005983 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.s.IndexSchema [deletereplica_test_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2005985 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.s.IndexSchema [deletereplica_test_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 2005987 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2005988 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'deletereplica_test_shard1_replica_n2' using configuration from collection deletereplica_test, trusted=true
   [junit4]   2> 2005988 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2005988 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'deletereplica_test_shard1_replica_n1' using configuration from collection deletereplica_test, trusted=true
   [junit4]   2> 2005988 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54682.solr.core.deletereplica_test.shard1.replica_n1' (registry 'solr.core.deletereplica_test.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2005988 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54683.solr.core.deletereplica_test.shard1.replica_n2' (registry 'solr.core.deletereplica_test.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@417adc80
   [junit4]   2> 2005988 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.c.SolrCore [[deletereplica_test_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-001/tempDir-001/node3/deletereplica_test_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-001/tempDir-001/node3/./deletereplica_test_shard1_replica_n1/data/]
   [junit4]   2> 2005988 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.c.SolrCore [[deletereplica_test_shard1_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-001/tempDir-001/node4/deletereplica_test_shard1_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-001/tempDir-001/node4/./deletereplica_test_shard1_replica_n2/data/]
   [junit4]   2> 2006051 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2006051 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2006051 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2006051 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2006053 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2006053 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2006053 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2006053 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2006054 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@3e582285[deletereplica_test_shard1_replica_n2] main]
   [junit4]   2> 2006054 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7d9098e5[deletereplica_test_shard1_replica_n1] main]
   [junit4]   2> 2006055 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2006055 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2006055 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2006055 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2006056 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2006056 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2006056 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1616634374630932480
   [junit4]   2> 2006056 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1616634374630932480
   [junit4]   2> 2006057 INFO  (searcherExecutor-10158-thread-1-processing-n:127.0.0.1:54683_solr x:deletereplica_test_shard1_replica_n2 c:deletereplica_test s:shard1 r:core_node4) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.c.SolrCore [deletereplica_test_shard1_replica_n2] Registered new searcher Searcher@3e582285[deletereplica_test_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2006057 INFO  (searcherExecutor-10159-thread-1-processing-n:127.0.0.1:54682_solr x:deletereplica_test_shard1_replica_n1 c:deletereplica_test s:shard1 r:core_node3) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.c.SolrCore [deletereplica_test_shard1_replica_n1] Registered new searcher Searcher@7d9098e5[deletereplica_test_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2006064 INFO  (qtp711349975-21155) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deletereplica_test/terms/shard1 to Terms{values={core_node4=0}, version=0}
   [junit4]   2> 2006064 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying
   [junit4]   2> 2006065 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deletereplica_test/terms/shard1 to Terms{values={core_node3=0, core_node4=0}, version=1}
   [junit4]   2> 2006071 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2006071 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2006071 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54682/solr/deletereplica_test_shard1_replica_n1/
   [junit4]   2> 2006072 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=deletereplica_test_shard1_replica_n1 url=http://127.0.0.1:54682/solr START replicas=[http://127.0.0.1:54683/solr/deletereplica_test_shard1_replica_n2/] nUpdates=100
   [junit4]   2> 2006072 INFO  (qtp17530944-21145) [n:127.0.0.1:54682_solr c:deletereplica_test s:shard1 r:core_node3 x:deletereplica_test_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=deletereplica_test_shard1_replica_n1 url=http://127.0.0.1:54682/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2006074 INFO  (qtp711349975-21166) [n:127.0.0.1:54683_solr c:deletereplica_test s:shard1 r:core_node4 x:deletereplica_test_shard1_replica_n2] o.a.s.c.S.Request [deletereplica_test_shard1_replica_n2]  webapp=/solr pa

[...truncated too long message...]

ncConnected type:NodeDataChanged path:/collections/deleteReplicaOnIndexing/state.json] for collection [deleteReplicaOnIndexing] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 329406 INFO  (zkCallback-185-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deletereplica_test/state.json] for collection [deletereplica_test] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 329470 INFO  (coreCloseExecutor-373-thread-1) [n:127.0.0.1:50659_solr c:raceDeleteReplica_true s:shard1 r:core_node4 x:raceDeleteReplica_true_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.raceDeleteReplica_true.shard1.leader, tag=272a9df8
   [junit4]   2> 329480 INFO  (coreCloseExecutor-374-thread-1) [n:127.0.0.1:50655_solr c:raceDeleteReplica_false s:shard1 r:core_node4 x:raceDeleteReplica_false_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.raceDeleteReplica_false.shard1.leader, tag=43b63214
   [junit4]   2> 329490 INFO  (coreCloseExecutor-372-thread-2) [n:127.0.0.1:50656_solr c:deleteReplicaOnIndexing s:shard1 r:core_node3 x:deleteReplicaOnIndexing_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.deleteReplicaOnIndexing.shard1.leader, tag=1f5b01ea
   [junit4]   2> 329490 INFO  (coreCloseExecutor-372-thread-2) [n:127.0.0.1:50656_solr c:deleteReplicaOnIndexing s:shard1 r:core_node3 x:deleteReplicaOnIndexing_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
   [junit4]   2> 329490 INFO  (coreCloseExecutor-372-thread-2) [n:127.0.0.1:50656_solr c:deleteReplicaOnIndexing s:shard1 r:core_node3 x:deleteReplicaOnIndexing_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@490c2517 commitCommandVersion:0
   [junit4]   2> 329509 INFO  (jetty-closer-128-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1bb4febe{/solr,null,UNAVAILABLE}
   [junit4]   2> 329511 INFO  (jetty-closer-128-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@cf838fb{/solr,null,UNAVAILABLE}
   [junit4]   2> 329513 INFO  (jetty-closer-128-thread-3) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 329514 INFO  (jetty-closer-128-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 329566 INFO  (jetty-closer-128-thread-2) [    ] o.a.s.c.Overseer Overseer (id=72080066361557007-127.0.0.1:50656_solr-n_0000000003) closing
   [junit4]   2> 329566 INFO  (OverseerStateUpdate-72080066361557007-127.0.0.1:50656_solr-n_0000000003) [n:127.0.0.1:50656_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:50656_solr
   [junit4]   2> 329567 WARN  (OverseerAutoScalingTriggerThread-72080066361557007-127.0.0.1:50656_solr-n_0000000003) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 329571 INFO  (jetty-closer-128-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2b5ccfef{/solr,null,UNAVAILABLE}
   [junit4]   2> 329571 INFO  (jetty-closer-128-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 329580 ERROR (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 329581 INFO  (SUITE-DeleteReplicaTest-seed#[6583D3DC3D2671FC]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50651 50651
   [junit4]   2> 329613 INFO  (Thread-366) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50651 50651
   [junit4]   2> 329768 WARN  (Thread-366) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 15 /solr/aliases.json
   [junit4]   2> 13 /solr/collections/raceDeleteReplica_true/terms/shard1
   [junit4]   2> 13 /solr/collections/deleteFromClusterState_true/terms/shard1
   [junit4]   2> 13 /solr/collections/deleteFromClusterState_false/terms/shard1
   [junit4]   2> 10 /solr/collections/deleteReplicaOnIndexing/terms/shard1
   [junit4]   2> 10 /solr/configs/conf
   [junit4]   2> 9 /solr/clusterprops.json
   [junit4]   2> 7 /solr/security.json
   [junit4]   2> 7 /solr/collections/deletereplica_test/terms/shard1
   [junit4]   2> 7 /solr/collections/raceDeleteReplica_false/terms/shard1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 17 /solr/collections/raceDeleteReplica_true/state.json
   [junit4]   2> 15 /solr/clusterprops.json
   [junit4]   2> 15 /solr/clusterstate.json
   [junit4]   2> 7 /solr/collections/raceDeleteReplica_false/state.json
   [junit4]   2> 5 /solr/collections/deleteFromClusterState_false/state.json
   [junit4]   2> 3 /solr/collections/deleteReplicaOnIndexing/state.json
   [junit4]   2> 3 /solr/overseer_elect/election/72080066361557000-127.0.0.1:50655_solr-n_0000000000
   [junit4]   2> 3 /solr/collections/deleteFromClusterState_true/state.json
   [junit4]   2> 2 /solr/collections/raceDeleteReplica_true/leader_elect/shard1/election/72080066361557002-core_node3-n_0000000000
   [junit4]   2> 2 /solr/collections/deleteFromClusterState_false/leader_elect/shard1/election/72080066361557021-core_node6-n_0000000000
   [junit4]   2> 2 /solr/autoscaling.json
   [junit4]   2> 2 /solr/collections/deletereplica_test/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/72080066361557013-127.0.0.1:50659_solr-n_0000000004
   [junit4]   2> 2 /solr/collections/raceDeleteReplica_false/leader_elect/shard1/election/72080066361557017-core_node3-n_0000000000
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 15 /solr/live_nodes
   [junit4]   2> 15 /solr/collections
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_6583D3DC3D2671FC-002
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {id=BlockTreeOrds(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=786, maxMBSortInHeap=6.412398590438496, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2078996f), locale=et, timezone=Asia/Katmandu
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=3,threads=1,free=96340824,total=298844160
   [junit4]   2> NOTE: All tests run in this JVM: [DeleteReplicaTest, DeleteReplicaTest]
   [junit4] Completed [5/5 (2!)] on J0 in 142.72s, 7 tests, 1 failure, 1 error, 2 skipped <<< FAILURES!
   [junit4]
   [junit4]
   [junit4] Tests with failures [seed: 6583D3DC3D2671FC]:
   [junit4]   - org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
   [junit4]   - org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards
   [junit4]   - org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest
   [junit4]   - org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
   [junit4]   - org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards
   [junit4]
   [junit4]
   [junit4] JVM J0:     1.13 ..   331.79 =   330.67s
   [junit4] JVM J1:     1.36 ..   263.61 =   262.26s
   [junit4] Execution time total: 5 minutes 31 seconds
   [junit4] Tests summary: 5 suites, 35 tests, 3 errors, 2 failures, 2 ignored (2 assumptions)

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1567: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1092: There were test failures: 5 suites, 35 tests, 3 errors, 2 failures, 2 ignored (2 assumptions) [seed: 6583D3DC3D2671FC]

Total time: 5 minutes 35 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   2/5 failed: org.apache.solr.cloud.DeleteReplicaTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=fbb987282e764aded31d7867e0d8f1fd6e2f02d4, workspace=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX
[WARNINGS] Computing warning deltas based on reference build #4919
Recording test results
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4921 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4921/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

5 tests failed.
FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:58676/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:58676/collection1
        at __randomizedtesting.SeedInfo.seed([5F64F2B58AEE3984:D730CD6F2412547C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:59684/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:59684/collection1
        at __randomizedtesting.SeedInfo.seed([5F64F2B58AEE3984:D730CD6F2412547C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:59687/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:59687/collection1
        at __randomizedtesting.SeedInfo.seed([5F64F2B58AEE3984:D730CD6F2412547C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:60691/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:60691/collection1
        at __randomizedtesting.SeedInfo.seed([5F64F2B58AEE3984:D730CD6F2412547C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:64956/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:64956/collection1
        at __randomizedtesting.SeedInfo.seed([5F64F2B58AEE3984:D730CD6F2412547C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more




Build Log:
[...truncated 14104 lines...]
   [junit4] Suite: org.apache.solr.cloud.RecoveryAfterSoftCommitTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/init-core-data-001
   [junit4]   2> 2319606 INFO  (SUITE-RecoveryAfterSoftCommitTest-seed#[5F64F2B58AEE3984]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2319610 INFO  (SUITE-RecoveryAfterSoftCommitTest-seed#[5F64F2B58AEE3984]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2319610 INFO  (SUITE-RecoveryAfterSoftCommitTest-seed#[5F64F2B58AEE3984]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2319610 INFO  (SUITE-RecoveryAfterSoftCommitTest-seed#[5F64F2B58AEE3984]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2319611 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2319612 INFO  (Thread-4447) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2319612 INFO  (Thread-4447) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2319615 ERROR (Thread-4447) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2319722 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.ZkTestServer start zk server on port:64935
   [junit4]   2> 2319734 INFO  (zkConnectionManagerCallback-8128-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2319738 INFO  (zkConnectionManagerCallback-8130-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2319748 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2319751 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2319753 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2319755 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2319757 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2319759 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2319762 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2319764 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2319766 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2319768 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2319771 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2319773 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 2320186 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2320188 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2320188 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2320188 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2320188 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6b66727d{/,null,AVAILABLE}
   [junit4]   2> 2320188 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@11b5e018{HTTP/1.1,[http/1.1]}{127.0.0.1:64939}
   [junit4]   2> 2320188 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.Server Started @2320269ms
   [junit4]   2> 2320188 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=64938, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/control-001/cores}
   [junit4]   2> 2320189 ERROR (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2320189 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2320189 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 2320189 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2320189 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2320189 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-09T18:15:22.773Z
   [junit4]   2> 2320191 INFO  (zkConnectionManagerCallback-8132-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2320192 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2320192 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/control-001/solr.xml
   [junit4]   2> 2320198 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2320198 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2320198 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2320311 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64935/solr
   [junit4]   2> 2320313 INFO  (zkConnectionManagerCallback-8136-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2320316 INFO  (zkConnectionManagerCallback-8138-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2320480 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [n:127.0.0.1:64938_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:64938_
   [junit4]   2> 2320482 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [n:127.0.0.1:64938_    ] o.a.s.c.Overseer Overseer (id=72082768345628676-127.0.0.1:64938_-n_0000000000) starting
   [junit4]   2> 2320493 INFO  (zkConnectionManagerCallback-8145-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2320496 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [n:127.0.0.1:64938_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64935/solr ready
   [junit4]   2> 2320497 INFO  (OverseerStateUpdate-72082768345628676-127.0.0.1:64938_-n_0000000000) [n:127.0.0.1:64938_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:64938_
   [junit4]   2> 2320497 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [n:127.0.0.1:64938_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64938_
   [junit4]   2> 2320499 INFO  (OverseerStateUpdate-72082768345628676-127.0.0.1:64938_-n_0000000000) [n:127.0.0.1:64938_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2320500 INFO  (zkCallback-8144-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2320539 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [n:127.0.0.1:64938_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2320560 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [n:127.0.0.1:64938_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8
   [junit4]   2> 2320566 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [n:127.0.0.1:64938_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8
   [junit4]   2> 2320566 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [n:127.0.0.1:64938_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8
   [junit4]   2> 2320567 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [n:127.0.0.1:64938_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/control-001/cores
   [junit4]   2> 2320596 INFO  (zkConnectionManagerCallback-8151-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2320598 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2320600 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64935/solr ready
   [junit4]   2> 2320601 INFO  (SocketProxy-Acceptor-64938) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=64945,localport=64938], receiveBufferSize:408300
   [junit4]   2> 2320605 INFO  (SocketProxy-Acceptor-64938) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=64939,localport=64946], receiveBufferSize=408300
   [junit4]   2> 2320607 INFO  (qtp1373590889-25868) [n:127.0.0.1:64938_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:64938_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2320611 INFO  (OverseerThreadFactory-10405-thread-1-processing-n:127.0.0.1:64938_) [n:127.0.0.1:64938_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2320730 INFO  (SocketProxy-Acceptor-64938) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=64947,localport=64938], receiveBufferSize:408300
   [junit4]   2> 2320730 INFO  (SocketProxy-Acceptor-64938) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=64939,localport=64948], receiveBufferSize=408300
   [junit4]   2> 2320734 INFO  (qtp1373590889-25864) [n:127.0.0.1:64938_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 2320739 INFO  (SocketProxy-Acceptor-64938) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=64949,localport=64938], receiveBufferSize:408300
   [junit4]   2> 2320740 INFO  (SocketProxy-Acceptor-64938) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=64939,localport=64950], receiveBufferSize=408300
   [junit4]   2> 2320741 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2320742 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2321775 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2321785 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2321805 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1119171188
   [junit4]   2> 2321865 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2321898 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2321898 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8
   [junit4]   2> 2321898 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2321900 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=36.017578125, floorSegmentMB=0.5576171875, forceMergeDeletesPctAllowed=5.030730978171832, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=31.68583918801062
   [junit4]   2> 2321902 WARN  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2321956 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2321956 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=2 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2321958 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2321958 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2321959 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=94.3837890625, floorSegmentMB=1.392578125, forceMergeDeletesPctAllowed=2.1232160231375143, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=48.795425698471014
   [junit4]   2> 2321959 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6d50dc1f[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2321961 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2321962 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2321962 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2321962 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1616681185623146496
   [junit4]   2> 2321963 INFO  (searcherExecutor-10410-thread-1-processing-n:127.0.0.1:64938_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@6d50dc1f[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2321971 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 2321977 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2321977 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2321977 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:64938/control_collection_shard1_replica_n1/
   [junit4]   2> 2321978 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2321978 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:64938/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2321984 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:64938/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2322141 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2322144 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1402
   [junit4]   2> 2322148 INFO  (qtp1373590889-25868) [n:127.0.0.1:64938_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2322254 INFO  (zkCallback-8137-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2322623 INFO  (OverseerCollectionConfigSetProcessor-72082768345628676-127.0.0.1:64938_-n_0000000000) [n:127.0.0.1:64938_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2323158 INFO  (qtp1373590889-25868) [n:127.0.0.1:64938_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:64938_&wt=javabin&version=2} status=0 QTime=2551
   [junit4]   2> 2323166 INFO  (zkConnectionManagerCallback-8156-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2323168 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2323169 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64935/solr ready
   [junit4]   2> 2323169 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2323170 INFO  (SocketProxy-Acceptor-64938) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=64954,localport=64938], receiveBufferSize:408300
   [junit4]   2> 2323170 INFO  (SocketProxy-Acceptor-64938) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=64939,localport=64955], receiveBufferSize=408300
   [junit4]   2> 2323171 INFO  (qtp1373590889-25870) [n:127.0.0.1:64938_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2323175 INFO  (OverseerThreadFactory-10405-thread-2-processing-n:127.0.0.1:64938_) [n:127.0.0.1:64938_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2323402 WARN  (OverseerThreadFactory-10405-thread-2-processing-n:127.0.0.1:64938_) [n:127.0.0.1:64938_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2323405 INFO  (qtp1373590889-25870) [n:127.0.0.1:64938_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2323405 INFO  (qtp1373590889-25870) [n:127.0.0.1:64938_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=233
   [junit4]   2> 2323784 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/shard-1-001 of type TLOG
   [junit4]   2> 2323785 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2323786 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2323786 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2323786 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2323786 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@66836c64{/,null,AVAILABLE}
   [junit4]   2> 2323787 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@64ad2105{HTTP/1.1,[http/1.1]}{127.0.0.1:64957}
   [junit4]   2> 2323787 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.Server Started @2323868ms
   [junit4]   2> 2323787 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=64956, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/shard-1-001/cores}
   [junit4]   2> 2323787 ERROR (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2323787 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2323787 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 2323787 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2323787 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2323787 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-09T18:15:26.371Z
   [junit4]   2> 2323789 INFO  (zkConnectionManagerCallback-8158-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2323790 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2323790 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/shard-1-001/solr.xml
   [junit4]   2> 2323795 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2323795 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2323796 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2323871 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64935/solr
   [junit4]   2> 2323872 INFO  (zkConnectionManagerCallback-8162-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2323875 INFO  (zkConnectionManagerCallback-8164-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2323888 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2323896 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2323896 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64956_
   [junit4]   2> 2323898 INFO  (zkCallback-8144-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2323898 INFO  (zkCallback-8137-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2323898 INFO  (zkCallback-8155-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2323898 INFO  (zkCallback-8163-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2323934 INFO  (zkConnectionManagerCallback-8171-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2323936 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2323938 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64935/solr ready
   [junit4]   2> 2323938 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2323960 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8
   [junit4]   2> 2323967 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8
   [junit4]   2> 2323967 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8
   [junit4]   2> 2323969 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/shard-1-001/cores
   [junit4]   2> 2323999 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:64956_&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2324002 INFO  (OverseerCollectionConfigSetProcessor-72082768345628676-127.0.0.1:64938_-n_0000000000) [n:127.0.0.1:64938_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2324008 INFO  (qtp1373590889-25870) [n:127.0.0.1:64938_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 2324009 INFO  (qtp1373590889-25866) [n:127.0.0.1:64938_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2324010 INFO  (SocketProxy-Acceptor-64956) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=64962,localport=64956], receiveBufferSize:408300
   [junit4]   2> 2324010 INFO  (SocketProxy-Acceptor-64956) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=64957,localport=64963], receiveBufferSize=408300
   [junit4]   2> 2324012 INFO  (qtp749929036-25931) [n:127.0.0.1:64956_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2324014 INFO  (OverseerThreadFactory-10405-thread-3-processing-n:127.0.0.1:64938_) [n:127.0.0.1:64938_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:64956_ for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 2324018 INFO  (SocketProxy-Acceptor-64956) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=64964,localport=64956], receiveBufferSize:408300
   [junit4]   2> 2324019 INFO  (SocketProxy-Acceptor-64956) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=64957,localport=64965], receiveBufferSize=408300
   [junit4]   2> 2324021 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_    x:collection1_shard1_replica_t21] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2325055 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2325064 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.IndexSchema [collection1_shard1_replica_t21] Schema name=test
   [junit4]   2> 2325144 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2325168 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t21' using configuration from collection collection1, trusted=true
   [junit4]   2> 2325169 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t21' (registry 'solr.core.collection1.shard1.replica_t21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8
   [junit4]   2> 2325169 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrCore [[collection1_shard1_replica_t21] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/shard-1-001/cores/collection1_shard1_replica_t21], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/shard-1-001/cores/collection1_shard1_replica_t21/data/]
   [junit4]   2> 2325170 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=36.017578125, floorSegmentMB=0.5576171875, forceMergeDeletesPctAllowed=5.030730978171832, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=31.68583918801062
   [junit4]   2> 2325171 WARN  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2325240 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2325240 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=2 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2325241 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2325241 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2325242 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=94.3837890625, floorSegmentMB=1.392578125, forceMergeDeletesPctAllowed=2.1232160231375143, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=48.795425698471014
   [junit4]   2> 2325242 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.SolrIndexSearcher Opening [Searcher@2ac88778[collection1_shard1_replica_t21] main]
   [junit4]   2> 2325243 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2325244 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2325244 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2325244 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1616681189064572928
   [junit4]   2> 2325245 INFO  (searcherExecutor-10424-thread-1-processing-n:127.0.0.1:64956_ x:collection1_shard1_replica_t21 c:collection1 s:shard1) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrCore [collection1_shard1_replica_t21] Registered new searcher Searcher@2ac88778[collection1_shard1_replica_t21] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2325252 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node22=0}, version=0}
   [junit4]   2> 2325258 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2325258 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2325258 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:64956/collection1_shard1_replica_t21/
   [junit4]   2> 2325259 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2325259 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SyncStrategy http://127.0.0.1:64956/collection1_shard1_replica_t21/ has no replicas
   [junit4]   2> 2325259 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ZkController collection1_shard1_replica_t21 stopping background replication from leader
   [junit4]   2> 2325265 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:64956/collection1_shard1_replica_t21/ shard1
   [junit4]   2> 2325427 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2325430 INFO  (qtp749929036-25927) [n:127.0.0.1:64956_ c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1410
   [junit4]   2> 2325434 INFO  (qtp1373590889-25867) [n:127.0.0.1:64938_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:64956_&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2} status=0 QTime=1434
   [junit4]   2> 2325530 INFO  (zkCallback-8163-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2325802 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/shard-2-001 of type TLOG
   [junit4]   2> 2325803 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2325804 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2325804 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2325804 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2325804 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@266b71c9{/,null,AVAILABLE}
   [junit4]   2> 2325805 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5c8c8af5{HTTP/1.1,[http/1.1]}{127.0.0.1:64967}
   [junit4]   2> 2325805 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.e.j.s.Server Started @2325885ms
   [junit4]   2> 2325805 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/tempDir-001/jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/, hostPort=64966, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/shard-2-001/cores}
   [junit4]   2> 2325805 ERROR (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2325805 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2325805 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 2325805 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2325805 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2325805 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-09T18:15:28.389Z
   [junit4]   2> 2325807 INFO  (zkConnectionManagerCallback-8174-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2325808 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2325808 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-001/shard-2-001/solr.xml
   [junit4]   2> 2325814 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2325814 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2325814 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3b3bfec8, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2325873 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64935/solr
   [junit4]   2> 2325875 INFO  (zkConnectionManagerCallback-8178-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2325878 INFO  (zkConnectionManagerCallback-8180-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2325890 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2325898 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2325898 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64966_
   [junit4]   2> 2325900 INFO  (zkCallback-8137-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2325900 INFO  (zkCallback-8144-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2325900 INFO  (zkCallback-8163-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2325900 INFO  (zkCallback-8179-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2325900 INFO  (zkCallback-8155-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2325901 INFO  (zkCallback-8170-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2325937 INFO  (zkConnectionManagerCallback-8187-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2325938 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F6

[...truncated too long message...]

ns:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/aliases.json
   [junit4]   2> 7 /solr/collections/collection1/terms/shard1
   [junit4]   2> 3 /solr/security.json
   [junit4]   2> 3 /solr/configs/conf1
   [junit4]   2> 2 /solr/collections/control_collection/terms/shard1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/clusterstate.json
   [junit4]   2> 8 /solr/clusterprops.json
   [junit4]   2> 6 /solr/collections/collection1/state.json
   [junit4]   2> 2 /solr/autoscaling.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/live_nodes
   [junit4]   2> 8 /solr/collections
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> 253898 WARN  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SocketProxy Closing 4 connections to: http://127.0.0.1:60675/, target: http://127.0.0.1:60676/
   [junit4]   2> 253898 WARN  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SocketProxy Closing 238 connections to: http://127.0.0.1:60691/, target: http://127.0.0.1:60692/
   [junit4]   2> 253899 WARN  (TEST-RecoveryAfterSoftCommitTest.test-seed#[5F64F2B58AEE3984]) [    ] o.a.s.c.SocketProxy Closing 3 connections to: http://127.0.0.1:60701/, target: http://127.0.0.1:60702/
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RecoveryAfterSoftCommitTest -Dtests.method=test -Dtests.seed=5F64F2B58AEE3984 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=es-SV -Dtests.timezone=America/Argentina/Catamarca -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   73.7s J0 | RecoveryAfterSoftCommitTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:60691/collection1
   [junit4]    > at __randomizedtesting.SeedInfo.seed([5F64F2B58AEE3984:D730CD6F2412547C]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
   [junit4]    > at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]    > at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]    > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]    > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]    > at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > ... 50 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_5F64F2B58AEE3984-003
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {multiDefault=Lucene50(blocksize=128), a_t=PostingsFormat(name=Asserting), id=PostingsFormat(name=LuceneVarGapFixedInterval), text=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Asserting), range_facet_i_dv=DocValuesFormat(name=Asserting), intDvoDefault=DocValuesFormat(name=Direct), timestamp=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=648, maxMBSortInHeap=6.198235381254417, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3f5c18ad), locale=es-SV, timezone=America/Argentina/Catamarca
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=3,threads=1,free=34667776,total=113979392
   [junit4]   2> NOTE: All tests run in this JVM: [RecoveryAfterSoftCommitTest, RecoveryAfterSoftCommitTest, RecoveryAfterSoftCommitTest]
   [junit4] Completed [5/5 (5!)] on J0 in 73.69s, 1 test, 1 error <<< FAILURES!
   [junit4]
   [junit4]
   [junit4] Tests with failures [seed: 5F64F2B58AEE3984]:
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]
   [junit4]
   [junit4] JVM J0:     0.70 ..   254.65 =   253.96s
   [junit4] JVM J1:     0.94 ..   182.27 =   181.33s
   [junit4] Execution time total: 4 minutes 14 seconds
   [junit4] Tests summary: 5 suites, 5 tests, 5 errors

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1567: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1092: There were test failures: 5 suites, 5 tests, 5 errors [seed: 5F64F2B58AEE3984]

Total time: 4 minutes 17 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   5/5 failed: org.apache.solr.cloud.RecoveryAfterSoftCommitTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=77a4bfaa90637cd3d9a8a2ef4889e163dab143aa, workspace=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX
[WARNINGS] Computing warning deltas based on reference build #4920
Recording test results
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4922 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4922/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

5 tests failed.
FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:51667/mm/hc/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:51667/mm/hc/collection1
        at __randomizedtesting.SeedInfo.seed([132FCEE6E81A1F:88471014481477E7]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:52683/mm/hc/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:52683/mm/hc/collection1
        at __randomizedtesting.SeedInfo.seed([132FCEE6E81A1F:88471014481477E7]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:52689/mm/hc/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:52689/mm/hc/collection1
        at __randomizedtesting.SeedInfo.seed([132FCEE6E81A1F:88471014481477E7]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:53690/mm/hc/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:53690/mm/hc/collection1
        at __randomizedtesting.SeedInfo.seed([132FCEE6E81A1F:88471014481477E7]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:58072/mm/hc/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:58072/mm/hc/collection1
        at __randomizedtesting.SeedInfo.seed([132FCEE6E81A1F:88471014481477E7]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
        at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 50 more




Build Log:
[...truncated 14101 lines...]
   [junit4] Suite: org.apache.solr.cloud.RecoveryAfterSoftCommitTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/init-core-data-001
   [junit4]   2> 2368356 INFO  (SUITE-RecoveryAfterSoftCommitTest-seed#[132FCEE6E81A1F]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /mm/hc
   [junit4]   2> 2368358 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2368358 INFO  (Thread-4058) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2368358 INFO  (Thread-4058) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2368362 ERROR (Thread-4058) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2368465 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.ZkTestServer start zk server on port:58036
   [junit4]   2> 2368476 INFO  (zkConnectionManagerCallback-6881-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2368480 INFO  (zkConnectionManagerCallback-6883-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2368489 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2368496 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2368509 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2368513 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2368516 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2368520 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2368524 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2368528 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2368532 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2368537 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2368540 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2368543 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 2373739 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2373740 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2373740 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2373740 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2373745 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1b64b875{/mm/hc,null,AVAILABLE}
   [junit4]   2> 2373748 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@39906681{HTTP/1.1,[http/1.1]}{127.0.0.1:58046}
   [junit4]   2> 2373748 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.Server Started @2373828ms
   [junit4]   2> 2373748 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/tempDir-001/control/data, replicaType=NRT, hostContext=/mm/hc, hostPort=58045, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/control-001/cores}
   [junit4]   2> 2373748 ERROR (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2373748 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2373749 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 2373749 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2373749 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2373749 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-10T06:14:01.365Z
   [junit4]   2> 2373751 INFO  (zkConnectionManagerCallback-6885-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2373754 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2373754 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/control-001/solr.xml
   [junit4]   2> 2373763 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2373764 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2373765 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2373845 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58036/solr
   [junit4]   2> 2373845 INFO  (zkConnectionManagerCallback-6889-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2373851 INFO  (zkConnectionManagerCallback-6891-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2374031 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58045_mm%2Fhc
   [junit4]   2> 2374032 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.Overseer Overseer (id=72085593850183684-127.0.0.1:58045_mm%2Fhc-n_0000000000) starting
   [junit4]   2> 2374044 INFO  (zkConnectionManagerCallback-6898-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2374047 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58036/solr ready
   [junit4]   2> 2374049 INFO  (OverseerStateUpdate-72085593850183684-127.0.0.1:58045_mm%2Fhc-n_0000000000) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:58045_mm%2Fhc
   [junit4]   2> 2374049 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58045_mm%2Fhc
   [junit4]   2> 2374052 INFO  (OverseerStateUpdate-72085593850183684-127.0.0.1:58045_mm%2Fhc-n_0000000000) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2374053 INFO  (zkCallback-6897-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2374097 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2374131 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378
   [junit4]   2> 2374153 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378
   [junit4]   2> 2374154 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378
   [junit4]   2> 2374157 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/control-001/cores
   [junit4]   2> 2374214 INFO  (zkConnectionManagerCallback-6904-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2374215 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2374217 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58036/solr ready
   [junit4]   2> 2374217 INFO  (SocketProxy-Acceptor-58045) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=58057,localport=58045], receiveBufferSize:408300
   [junit4]   2> 2374217 INFO  (SocketProxy-Acceptor-58045) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58046,localport=58058], receiveBufferSize=408300
   [junit4]   2> 2374222 INFO  (qtp760841123-24100) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:58045_mm%252Fhc&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2374225 INFO  (OverseerThreadFactory-9901-thread-1-processing-n:127.0.0.1:58045_mm%2Fhc) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2374333 INFO  (SocketProxy-Acceptor-58045) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=58061,localport=58045], receiveBufferSize:408300
   [junit4]   2> 2374334 INFO  (SocketProxy-Acceptor-58045) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58046,localport=58062], receiveBufferSize=408300
   [junit4]   2> 2374337 INFO  (qtp760841123-24096) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=3
   [junit4]   2> 2374343 INFO  (SocketProxy-Acceptor-58045) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=58063,localport=58045], receiveBufferSize:408300
   [junit4]   2> 2374344 INFO  (SocketProxy-Acceptor-58045) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58046,localport=58064], receiveBufferSize=408300
   [junit4]   2> 2374344 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2374345 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2375378 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2375387 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2375476 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2375503 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2375503 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378
   [junit4]   2> 2375504 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2375506 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=0.1416015625, floorSegmentMB=1.7568359375, forceMergeDeletesPctAllowed=20.59121710651324, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=23.425662296584974
   [junit4]   2> 2375507 WARN  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2375572 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2375572 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=2 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2375574 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2375574 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2375577 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.5517578125, noCFSRatio=0.8479371396139758]
   [junit4]   2> 2375593 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@19376990[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2375594 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2375595 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2375595 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2375596 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1616726398821466112
   [junit4]   2> 2375596 INFO  (searcherExecutor-9906-thread-1-processing-n:127.0.0.1:58045_mm%2Fhc x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@19376990[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2375604 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 2375609 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2375610 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2375610 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:58045/mm/hc/control_collection_shard1_replica_n1/
   [junit4]   2> 2375610 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2375611 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:58045/mm/hc/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2375617 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:58045/mm/hc/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2375722 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2375725 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1380
   [junit4]   2> 2375728 INFO  (qtp760841123-24100) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2375828 INFO  (zkCallback-6890-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2376228 INFO  (OverseerCollectionConfigSetProcessor-72085593850183684-127.0.0.1:58045_mm%2Fhc-n_0000000000) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2376729 INFO  (qtp760841123-24100) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:58045_mm%252Fhc&wt=javabin&version=2} status=0 QTime=2507
   [junit4]   2> 2376735 INFO  (zkConnectionManagerCallback-6909-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2376737 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2376739 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58036/solr ready
   [junit4]   2> 2376739 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2376739 INFO  (SocketProxy-Acceptor-58045) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=58068,localport=58045], receiveBufferSize:408300
   [junit4]   2> 2376740 INFO  (SocketProxy-Acceptor-58045) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58046,localport=58069], receiveBufferSize=408300
   [junit4]   2> 2376740 INFO  (qtp760841123-24102) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2376744 INFO  (OverseerThreadFactory-9901-thread-2-processing-n:127.0.0.1:58045_mm%2Fhc) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2376952 WARN  (OverseerThreadFactory-9901-thread-2-processing-n:127.0.0.1:58045_mm%2Fhc) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2376955 INFO  (qtp760841123-24102) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2376955 INFO  (qtp760841123-24102) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=214
   [junit4]   2> 2377398 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/shard-1-001 of type TLOG
   [junit4]   2> 2377399 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2377400 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2377400 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2377400 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2377400 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2d565033{/mm/hc,null,AVAILABLE}
   [junit4]   2> 2377401 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@66ee301f{HTTP/1.1,[http/1.1]}{127.0.0.1:58073}
   [junit4]   2> 2377401 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.Server Started @2377481ms
   [junit4]   2> 2377401 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/mm/hc, hostPort=58072, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/shard-1-001/cores}
   [junit4]   2> 2377401 ERROR (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2377401 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2377402 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 2377402 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2377402 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2377402 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-10T06:14:05.018Z
   [junit4]   2> 2377404 INFO  (zkConnectionManagerCallback-6911-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2377405 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2377405 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/shard-1-001/solr.xml
   [junit4]   2> 2377436 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2377436 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2377437 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2377874 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58036/solr
   [junit4]   2> 2377876 INFO  (zkConnectionManagerCallback-6915-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2377879 INFO  (zkConnectionManagerCallback-6917-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2377892 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2377900 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2377900 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58072_mm%2Fhc
   [junit4]   2> 2377901 INFO  (zkCallback-6890-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2377901 INFO  (zkCallback-6897-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2377901 INFO  (zkCallback-6908-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2377901 INFO  (zkCallback-6916-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2377959 INFO  (zkConnectionManagerCallback-6924-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2377961 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2377961 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58036/solr ready
   [junit4]   2> 2377966 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2377991 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378
   [junit4]   2> 2378000 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378
   [junit4]   2> 2378000 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378
   [junit4]   2> 2378001 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/shard-1-001/cores
   [junit4]   2> 2378056 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:58072_mm%252Fhc&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2378060 INFO  (OverseerCollectionConfigSetProcessor-72085593850183684-127.0.0.1:58045_mm%2Fhc-n_0000000000) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2378064 INFO  (qtp760841123-24102) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 2378066 INFO  (qtp760841123-24098) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2378067 INFO  (SocketProxy-Acceptor-58072) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=58085,localport=58072], receiveBufferSize:408300
   [junit4]   2> 2378068 INFO  (SocketProxy-Acceptor-58072) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58073,localport=58086], receiveBufferSize=408300
   [junit4]   2> 2378069 INFO  (qtp1553530712-24164) [n:127.0.0.1:58072_mm%2Fhc    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2378070 INFO  (OverseerThreadFactory-9901-thread-3-processing-n:127.0.0.1:58045_mm%2Fhc) [n:127.0.0.1:58045_mm%2Fhc c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:58072_mm%2Fhc for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 2378076 INFO  (SocketProxy-Acceptor-58072) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=58087,localport=58072], receiveBufferSize:408300
   [junit4]   2> 2378077 INFO  (SocketProxy-Acceptor-58072) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58073,localport=58088], receiveBufferSize=408300
   [junit4]   2> 2378079 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc    x:collection1_shard1_replica_t21] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2379106 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2379115 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.IndexSchema [collection1_shard1_replica_t21] Schema name=test
   [junit4]   2> 2379220 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2379248 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t21' using configuration from collection collection1, trusted=true
   [junit4]   2> 2379248 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t21' (registry 'solr.core.collection1.shard1.replica_t21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378
   [junit4]   2> 2379249 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrCore [[collection1_shard1_replica_t21] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/shard-1-001/cores/collection1_shard1_replica_t21], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/shard-1-001/cores/collection1_shard1_replica_t21/data/]
   [junit4]   2> 2379250 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=0.1416015625, floorSegmentMB=1.7568359375, forceMergeDeletesPctAllowed=20.59121710651324, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=23.425662296584974
   [junit4]   2> 2379251 WARN  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2379327 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2379327 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=2 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2379328 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2379328 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2379328 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.5517578125, noCFSRatio=0.8479371396139758]
   [junit4]   2> 2379329 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.SolrIndexSearcher Opening [Searcher@121c089[collection1_shard1_replica_t21] main]
   [junit4]   2> 2379330 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2379331 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2379331 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2379331 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1616726402737897472
   [junit4]   2> 2379332 INFO  (searcherExecutor-9920-thread-1-processing-n:127.0.0.1:58072_mm%2Fhc x:collection1_shard1_replica_t21 c:collection1 s:shard1) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SolrCore [collection1_shard1_replica_t21] Registered new searcher Searcher@121c089[collection1_shard1_replica_t21] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2379339 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node22=0}, version=0}
   [junit4]   2> 2379343 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2379344 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2379344 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:58072/mm/hc/collection1_shard1_replica_t21/
   [junit4]   2> 2379344 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2379344 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.SyncStrategy http://127.0.0.1:58072/mm/hc/collection1_shard1_replica_t21/ has no replicas
   [junit4]   2> 2379344 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ZkController collection1_shard1_replica_t21 stopping background replication from leader
   [junit4]   2> 2379349 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:58072/mm/hc/collection1_shard1_replica_t21/ shard1
   [junit4]   2> 2379456 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2379459 INFO  (qtp1553530712-24160) [n:127.0.0.1:58072_mm%2Fhc c:collection1 s:shard1  x:collection1_shard1_replica_t21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1380
   [junit4]   2> 2379462 INFO  (qtp760841123-24099) [n:127.0.0.1:58045_mm%2Fhc c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:58072_mm%252Fhc&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2} status=0 QTime=1406
   [junit4]   2> 2380056 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/shard-2-001 of type TLOG
   [junit4]   2> 2380056 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2380057 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2380057 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2380057 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2380058 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@12caa905{/mm/hc,null,AVAILABLE}
   [junit4]   2> 2380058 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@224133fe{HTTP/1.1,[http/1.1]}{127.0.0.1:58092}
   [junit4]   2> 2380058 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.e.j.s.Server Started @2380138ms
   [junit4]   2> 2380058 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/tempDir-001/jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/mm/hc, hostPort=58091, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/shard-2-001/cores}
   [junit4]   2> 2380059 ERROR (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2380059 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2380059 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 2380059 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2380059 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2380060 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-10T06:14:07.676Z
   [junit4]   2> 2380061 INFO  (zkConnectionManagerCallback-6927-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2380063 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2380063 INFO  (OverseerCollectionConfigSetProcessor-72085593850183684-127.0.0.1:58045_mm%2Fhc-n_0000000000) [n:127.0.0.1:58045_mm%2Fhc    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2380063 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-001/shard-2-001/solr.xml
   [junit4]   2> 2380069 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2380069 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2380070 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5bc39378, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2380165 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58036/solr
   [junit4]   2> 2380168 INFO  (zkConnectionManagerCallback-6931-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2380172 INFO  (zkConnectionManagerCallback-6933-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2380185 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2380193 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2380193 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58091_mm%2Fhc
   [junit4]   2> 2380199 INFO  (zkCallback-6890-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2380199 INFO  (zkCallback-6923-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2380199 INFO  (zkCallback-6908-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2380199 INFO  (zkCallback-6897-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2380199 INFO  (zkCallback-6916-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2380200 INFO  (zkCallback-6932-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2380241 INFO  (zkConnectionManagerCallback-6940-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2380246 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2380248 INFO  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58036/solr ready
   [junit4]   2> 2380249 INFO  (T

[...truncated too long message...]

on1/terms/shard1
   [junit4]   2> 4 /solr/collections/collection1/state.json
   [junit4]   2> 3 /solr/security.json
   [junit4]   2> 3 /solr/configs/conf1
   [junit4]   2> 2 /solr/collections/control_collection/terms/shard1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/clusterstate.json
   [junit4]   2> 8 /solr/clusterprops.json
   [junit4]   2> 2 /solr/autoscaling.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/live_nodes
   [junit4]   2> 8 /solr/collections
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 2 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 2 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> 252193 WARN  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SocketProxy Closing 3 connections to: http://127.0.0.1:53700/mm/hc, target: http://127.0.0.1:53701/mm/hc
   [junit4]   2> 252193 WARN  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SocketProxy Closing 4 connections to: http://127.0.0.1:53674/mm/hc, target: http://127.0.0.1:53675/mm/hc
   [junit4]   2> 252193 WARN  (TEST-RecoveryAfterSoftCommitTest.test-seed#[132FCEE6E81A1F]) [    ] o.a.s.c.SocketProxy Closing 239 connections to: http://127.0.0.1:53690/mm/hc, target: http://127.0.0.1:53691/mm/hc
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RecoveryAfterSoftCommitTest -Dtests.method=test -Dtests.seed=132FCEE6E81A1F -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=es-PE -Dtests.timezone=PST8PDT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   75.0s J1 | RecoveryAfterSoftCommitTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:53690/mm/hc/collection1
   [junit4]    > at __randomizedtesting.SeedInfo.seed([132FCEE6E81A1F:88471014481477E7]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
   [junit4]    > at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:87)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]    > at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]    > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]    > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
   [junit4]    > at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > ... 50 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.RecoveryAfterSoftCommitTest_132FCEE6E81A1F-003
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {range_facet_l_dv=FSTOrd50, multiDefault=FST50, a_t=FST50, intDefault=PostingsFormat(name=LuceneFixedGap), id_i1=PostingsFormat(name=LuceneFixedGap), id=FSTOrd50, range_facet_i_dv=FST50, text=Lucene50(blocksize=128), range_facet_l=FST50, timestamp=FST50}, docValues:{range_facet_l_dv=DocValuesFormat(name=Direct), _version_=DocValuesFormat(name=Asserting), intDefault=DocValuesFormat(name=Asserting), id_i1=DocValuesFormat(name=Asserting), range_facet_i_dv=DocValuesFormat(name=Lucene70), intDvoDefault=DocValuesFormat(name=Direct), range_facet_l=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=947, maxMBSortInHeap=7.877502039382886, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@41c2ec82), locale=es-PE, timezone=PST8PDT
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=3,threads=1,free=41694696,total=113106944
   [junit4]   2> NOTE: All tests run in this JVM: [RecoveryAfterSoftCommitTest, RecoveryAfterSoftCommitTest, RecoveryAfterSoftCommitTest]
   [junit4] Completed [5/5 (5!)] on J1 in 75.02s, 1 test, 1 error <<< FAILURES!
   [junit4]
   [junit4]
   [junit4] Tests with failures [seed: 132FCEE6E81A1F]:
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]   - org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
   [junit4]
   [junit4]
   [junit4] JVM J0:     0.69 ..   177.53 =   176.84s
   [junit4] JVM J1:     0.93 ..   253.25 =   252.32s
   [junit4] Execution time total: 4 minutes 13 seconds
   [junit4] Tests summary: 5 suites, 5 tests, 5 errors

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1567: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1092: There were test failures: 5 suites, 5 tests, 5 errors [seed: 132FCEE6E81A1F]

Total time: 4 minutes 15 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   5/5 failed: org.apache.solr.cloud.RecoveryAfterSoftCommitTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=42ee96642c01fb6280c5af161761ff28be6ee914, workspace=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX
[WARNINGS] Computing warning deltas based on reference build #4921
Recording test results
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4923 - Failure!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4923/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 15173 lines...]
   [junit4] JVM J1: stdout was not empty, see: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/temp/junit4-J1-20181110_190126_32514878801079108887242.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] #
   [junit4] # A fatal error has been detected by the Java Runtime Environment:
   [junit4] #
   [junit4] #  SIGSEGV (0xb) at pc=0x000000010b5329d8, pid=74095, tid=17155
   [junit4] #
   [junit4] # JRE version: Java(TM) SE Runtime Environment (9.0+181) (build 9+181)
   [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (9+181, mixed mode, tiered, compressed oops, parallel gc, bsd-amd64)
   [junit4] # Problematic frame:
   [junit4] # V  [libjvm.dylib+0x4f49d8]  Node::in(unsigned int) const+0x18
   [junit4] #
   [junit4] # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
   [junit4] #
   [junit4] # An error report file with more information is saved as:
   [junit4] # /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/hs_err_pid74095.log
   [junit4] #
   [junit4] # Compiler replay data is saved as:
   [junit4] # /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/replay_pid74095.log
   [junit4] #
   [junit4] # If you would like to submit a bug report, please visit:
   [junit4] #   http://bugreport.java.com/bugreport/crash.jsp
   [junit4] #
   [junit4] <<< JVM J1: EOF ----

[...truncated 287 lines...]
   [junit4] ERROR: JVM J1 ended with an exception, command line: /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home/bin/java -XX:+UseCompressedOops -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/heapdumps -ea -esa --illegal-access=deny -Dtests.prefix=tests -Dtests.seed=FEED3CCC9F9AD38A -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=1 -DtempDir=./temp -Djava.io.tmpdir=./temp -Dcommon.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene -Dclover.db.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/clover/db -Djava.security.policy=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/tools/junit4/solr-tests.policy -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.src.home=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX -Djava.security.egd=file:/dev/./urandom -Djunit4.childvm.cwd=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1 -Djunit4.tempDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/temp -Djunit4.childvm.id=1 -Djunit4.childvm.count=2 -Dfile.encoding=ISO-8859-1 -Dtests.disableHdfs=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dtests.filterstacks=true -Dtests.leaveTemporary=false -Dtests.badapples=false -classpath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-test-framework/classes/java:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/test-framework/lib/junit4-ant-2.6.4.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/test-framework/classes/java:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/codecs/classes/java:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/classes/java:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/java:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/common/lucene-analyzers-common-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/nori/lucene-analyzers-nori-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/codecs/lucene-codecs-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/backward-codecs/lucene-backward-codecs-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/highlighter/lucene-highlighter-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/memory/lucene-memory-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/misc/lucene-misc-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/spatial-extras/lucene-spatial-extras-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/spatial3d/lucene-spatial3d-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/expressions/lucene-expressions-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/suggest/lucene-suggest-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/grouping/lucene-grouping-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/queries/lucene-queries-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/queryparser/lucene-queryparser-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/join/lucene-join-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/sandbox/lucene-sandbox-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/classification/lucene-classification-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/antlr4-runtime-4.5.1-1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/asm-5.1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/asm-commons-5.1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/avatica-core-1.10.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/caffeine-2.4.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/calcite-core-1.13.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/calcite-linq4j-1.13.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-cli-1.2.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-codec-1.11.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-collections-3.2.2.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-compiler-2.7.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-configuration-1.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-exec-1.3.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-fileupload-1.3.3.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-io-2.5.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-lang-2.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-lang3-3.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/commons-math3-3.6.1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/curator-client-2.8.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/curator-framework-2.8.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/curator-recipes-2.8.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/disruptor-3.4.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/dom4j-1.6.1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/eigenbase-properties-1.1.5.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/guava-14.0.1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/hadoop-annotations-2.7.4.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/hadoop-auth-2.7.4.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/hadoop-common-2.7.4.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/hadoop-hdfs-2.7.4.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/hppc-0.8.1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/htrace-core-3.2.0-incubating.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/jackson-annotations-2.9.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/jackson-core-2.9.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/jackson-core-asl-1.9.13.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/jackson-databind-2.9.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/jackson-dataformat-smile-2.9.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/jackson-mapper-asl-1.9.13.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/janino-2.7.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/jcl-over-slf4j-1.7.24.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/log4j-1.2-api-2.11.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/log4j-api-2.11.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/log4j-core-2.11.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/log4j-slf4j-impl-2.11.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/org.restlet-2.3.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/org.restlet.ext.servlet-2.3.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/protobuf-java-3.1.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/rrd4j-3.2.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/spatial4j-0.7.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/lib/t-digest-3.1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/commons-io-2.5.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/commons-math3-3.6.1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/httpclient-4.5.3.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/httpcore-4.4.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/httpmime-4.5.3.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/jcl-over-slf4j-1.7.24.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/noggit-0.8.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/slf4j-api-1.7.24.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/stax2-api-3.1.4.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solrj/lib/zookeeper-3.4.11.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/gmetric4j-1.0.7.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/javax.servlet-api-3.1.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-continuation-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-deploy-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-http-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-io-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-jmx-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-rewrite-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-security-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-server-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-servlet-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-servlets-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-util-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-webapp-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/jetty-xml-9.4.11.v20180605.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/metrics-core-3.2.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/metrics-ganglia-3.2.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/metrics-graphite-3.2.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/metrics-jetty9-3.2.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/server/lib/metrics-jvm-3.2.6.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/example/example-DIH/solr/db/lib/hsqldb-2.4.0.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/core/classes/java:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/test-framework/lib/junit-4.10.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/test-framework/lib/randomizedtesting-runner-2.6.4.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/antlr-2.7.7.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/antlr4-runtime-4.5.1-1.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-core-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-core-api-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-core-avl-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-core-shared-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-i18n-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptor-kerberos-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/api-all-1.0.0-M20.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/bcprov-jdk15on-1.60.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/byte-buddy-1.6.2.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/commons-collections-3.2.2.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/ehcache-core-2.4.4.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/jersey-core-1.9.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/jersey-server-1.9.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/jetty-6.1.26.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/jetty-util-6.1.26.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/mina-core-2.0.0-M5.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/mockito-core-2.6.2.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/netty-all-4.0.36.Final.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/test-lib/objenesis-2.5.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/analysis/icu/lucene-analyzers-icu-8.0.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-analysis-extras/classes/java:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/queryparser/classes/test:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/backward-codecs/classes/test:/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/contrib/analysis-extras/lib/icu4j-62.1.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-launcher.jar:/Users/jenkins/.ant/lib/ivy-2.4.0.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-antlr.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bcel.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bsf.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-log4j.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-oro.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-regexp.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-resolver.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-xalan2.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-logging.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-net.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jai.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-javamail.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jdepend.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jmf.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jsch.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit4.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-netrexx.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-swing.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-testutil.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant.jar:/Users/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.6.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/temp/junit4-J1-20181110_190126_32513279676883356824656.events @/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/temp/junit4-J1-20181110_190126_3251640810171627989481.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Forked process returned with error code: 134. Very likely a JVM crash.  See process stdout at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/temp/junit4-J1-20181110_190126_32514878801079108887242.sysout
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1542)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:123)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:997)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:994)
   [junit4] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   [junit4] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
   [junit4] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
   [junit4] at java.base/java.lang.Thread.run(Thread.java:844)

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:633: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:577: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:59: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build.xml:267: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/common-build.xml:558: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1567: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1092: At least one slave process threw an exception, first: Forked process returned with error code: 134. Very likely a JVM crash.  See process stdout at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/temp/junit4-J1-20181110_190126_32514878801079108887242.sysout

Total time: 83 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4924 - Still unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4924/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica

Error Message:
Timeout waiting for collection to become active Live Nodes: [127.0.0.1:10006_solr, 127.0.0.1:10008_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr, 127.0.0.1:10007_solr] Last available state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/34)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "policy":"c1",   "shards":{"shard1":{       "replicas":{"core_node1":{           "core":"testCreateCollectionAddReplica_shard1_replica_n1",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10005_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"80000000-7fffffff",       "state":"active"}}}

Stack Trace:
java.lang.AssertionError: Timeout waiting for collection to become active
Live Nodes: [127.0.0.1:10006_solr, 127.0.0.1:10008_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr, 127.0.0.1:10007_solr]
Last available state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/34)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":"testCreateCollectionAddReplica_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10005_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}
        at __randomizedtesting.SeedInfo.seed([D59B683037746B7A:55BB0D1E263783DC]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)
        at org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica(TestSimPolicyCloud.java:123)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 35
live nodes:[127.0.0.1:10006_solr, 127.0.0.1:10008_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr, 127.0.0.1:10007_solr]
collections:{testCreateCollectionAddReplica=DocCollection(testCreateCollectionAddReplica//clusterstate.json/34)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":"testCreateCollectionAddReplica_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10005_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}}, last coll state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/34)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":"testCreateCollectionAddReplica_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10005_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:112)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:64)
        ... 41 more


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica

Error Message:
Timeout waiting for collection to become active Live Nodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr] Last available state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/23)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "policy":"c1",   "shards":{"shard1":{       "replicas":{"core_node1":{           "core":"testCreateCollectionAddReplica_shard1_replica_n1",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10004_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"80000000-7fffffff",       "state":"active"}}}

Stack Trace:
java.lang.AssertionError: Timeout waiting for collection to become active
Live Nodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
Last available state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/23)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":"testCreateCollectionAddReplica_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10004_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}
        at __randomizedtesting.SeedInfo.seed([D59B683037746B7A:55BB0D1E263783DC]:0)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)
        at org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica(TestSimPolicyCloud.java:123)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 24
live nodes:[127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
collections:{testCreateCollectionAddReplica=DocCollection(testCreateCollectionAddReplica//clusterstate.json/23)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":"testCreateCollectionAddReplica_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10004_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}}, last coll state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/23)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
      "replicas":{"core_node1":{
          "core":"testCreateCollectionAddReplica_shard1_replica_n1",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":10240,
          "node_name":"127.0.0.1:10004_solr",
          "state":"active",
          "type":"NRT",
          "INDEX.sizeInGB":9.5367431640625E-6,
          "SEARCHER.searcher.numDocs":0}},
      "range":"80000000-7fffffff",
      "state":"active"}}}
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:112)
        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:64)
        ... 41 more




Build Log:
[...truncated 12779 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud
   [junit4]   2> 550801 INFO  (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimPolicyCloud_D59B683037746B7A-001/init-core-data-001
   [junit4]   2> 550802 INFO  (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 550806 INFO  (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 550849 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 550849 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 550849 DEBUG (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10000_solr
   [junit4]   2> 550849 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 550849 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 550852 DEBUG (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=0
   [junit4]   2> 550855 DEBUG (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
   [junit4]   2> 550855 INFO  (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 550855 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10000_solr]
   [junit4]   2> 550856 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 550856 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 550858 DEBUG (ScheduledTrigger-2608-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 550859 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateCollectionAddShardWithReplicaTypeUsingPolicy
   [junit4]   2> 550860 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 550860 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 550861 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 550861 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 550861 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 550862 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 550863 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 550863 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 550863 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 550863 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 550864 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 550864 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 550865 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 550865 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 550866 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 550866 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 550866 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 550867 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 550867 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 550867 DEBUG (ScheduledTrigger-2613-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 550887 DEBUG (ScheduledTrigger-2613-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 550908 DEBUG (ScheduledTrigger-2613-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 550935 DEBUG (ScheduledTrigger-2613-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 550955 DEBUG (ScheduledTrigger-2613-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 550976 DEBUG (ScheduledTrigger-2613-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 550996 DEBUG (ScheduledTrigger-2613-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551016 DEBUG (ScheduledTrigger-2613-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551037 DEBUG (ScheduledTrigger-2613-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551058 DEBUG (ScheduledTrigger-2613-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551066 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.TestSimPolicyCloud NRT 127.0.0.1:10001_solr PULL 127.0.0.1:10004_solr , TLOG 127.0.0.1:10000_solr
   [junit4]   2> 551070 DEBUG (simCloudManagerPool-2612-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=1
   [junit4]   2> 551070 DEBUG (simCloudManagerPool-2612-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 1
   [junit4]   2> 551072 DEBUG (simCloudManagerPool-2612-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 551072 DEBUG (simCloudManagerPool-2612-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 551073 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 551073 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 551073 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 551075 DEBUG (simCloudManagerPool-2612-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection policiesTest, currentVersion=2
   [junit4]   2> 551075 DEBUG (simCloudManagerPool-2612-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=2
   [junit4]   2> 551075 DEBUG (simCloudManagerPool-2612-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 2
   [junit4]   2> 551088 DEBUG (ScheduledTrigger-2613-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551097 DEBUG (simCloudManagerPool-2612-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=3
   [junit4]   2> 551100 DEBUG (simCloudManagerPool-2612-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 3
   [junit4]   2> 551108 DEBUG (simCloudManagerPool-2612-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection policiesTest, currentVersion=4
   [junit4]   2> 551111 DEBUG (ScheduledTrigger-2613-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551134 DEBUG (ScheduledTrigger-2613-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551137 DEBUG (simCloudManagerPool-2612-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=4
   [junit4]   2> 551138 DEBUG (simCloudManagerPool-2612-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 4
   [junit4]   2> 551142 DEBUG (simCloudManagerPool-2612-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for policiesTest / s1 (currentVersion=5): {"core_node1":{
   [junit4]   2>     "core":"policiesTest_s1_replica_n1",
   [junit4]   2>     "shard":"s1",
   [junit4]   2>     "collection":"policiesTest",
   [junit4]   2>     "node_name":"127.0.0.1:10001_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 551153 DEBUG (simCloudManagerPool-2612-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=5
   [junit4]   2> 551153 DEBUG (simCloudManagerPool-2612-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 5
   [junit4]   2> 551154 DEBUG (ScheduledTrigger-2613-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551165 DEBUG (simCloudManagerPool-2612-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=6
   [junit4]   2> 551166 DEBUG (simCloudManagerPool-2612-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 6
   [junit4]   2> 551166 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=7
   [junit4]   2> 551167 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 7
   [junit4]   2> 551182 DEBUG (ScheduledTrigger-2613-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551204 DEBUG (ScheduledTrigger-2613-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551226 DEBUG (ScheduledTrigger-2613-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551256 DEBUG (ScheduledTrigger-2613-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551280 INFO  (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10001_solr for creating new replica of shard s3 for collection policiesTest
   [junit4]   2> 551282 DEBUG (ScheduledTrigger-2613-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551292 DEBUG (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=8
   [junit4]   2> 551293 DEBUG (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 8
   [junit4]   2> 551293 INFO  (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10000_solr for creating new replica of shard s3 for collection policiesTest
   [junit4]   2> 551294 DEBUG (simCloudManagerPool-2612-thread-8) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for policiesTest / s3 (currentVersion=9): {"core_node5":{
   [junit4]   2>     "core":"policiesTest_s3_replica_n4",
   [junit4]   2>     "shard":"s3",
   [junit4]   2>     "collection":"policiesTest",
   [junit4]   2>     "node_name":"127.0.0.1:10001_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "pullReplicas":"1",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "nrtReplicas":"1",
   [junit4]   2>     "tlogReplicas":"1",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6}}
   [junit4]   2> 551308 DEBUG (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=9
   [junit4]   2> 551309 DEBUG (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 9
   [junit4]   2> 551309 INFO  (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10004_solr for creating new replica of shard s3 for collection policiesTest
   [junit4]   2> 551309 DEBUG (ScheduledTrigger-2613-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551320 DEBUG (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=10
   [junit4]   2> 551320 DEBUG (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 10
   [junit4]   2> 551320 DEBUG (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=11
   [junit4]   2> 551321 DEBUG (simCloudManagerPool-2612-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 11
   [junit4]   2> 551321 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateCollectionAddShardWithReplicaTypeUsingPolicy
   [junit4]   2> 551322 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=12
   [junit4]   2> 551323 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 12
   [junit4]   2> 551332 DEBUG (ScheduledTrigger-2613-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551353 DEBUG (ScheduledTrigger-2613-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551380 DEBUG (ScheduledTrigger-2613-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551405 DEBUG (ScheduledTrigger-2613-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551436 DEBUG (ScheduledTrigger-2613-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551455 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 5
   [junit4]   2> ## Empty nodes: 2
   [junit4]   2> ## Dead nodes: 0
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * policiesTest
   [junit4]   2> ##    shardsTotal 2
   [junit4]   2> ##    shardsState {active=2}
   [junit4]   2> ##      shardsWithoutLeader 0
   [junit4]   2> ##    totalReplicas 6
   [junit4]   2> ##      activeReplicas 6
   [junit4]   2> ##      inactiveReplicas 0
   [junit4]   2> ##    totalActiveDocs 0
   [junit4]   2> ##      bufferedDocs 0
   [junit4]   2> ##      maxActiveSliceDocs 0
   [junit4]   2> ##      minActiveSliceDocs 0
   [junit4]   2> ##      avgActiveSliceDocs 0
   [junit4]   2> ##    totalInactiveDocs 0
   [junit4]   2> ##      maxInactiveSliceDocs 0
   [junit4]   2> ##      minInactiveSliceDocs 0
   [junit4]   2> ##      avgInactiveSliceDocs NaN
   [junit4]   2> ##    totalActiveBytes 20,480
   [junit4]   2> ##      maxActiveSliceBytes 10,240
   [junit4]   2> ##      minActiveSliceBytes 10,240
   [junit4]   2> ##      avgActiveSliceBytes 10,240
   [junit4]   2> ##    totalInactiveBytes 0
   [junit4]   2> ##      maxInactiveSliceBytes 0
   [junit4]   2> ##      minInactiveSliceBytes 0
   [junit4]   2> ##      avgInactiveSliceBytes NaN
   [junit4]   2> ##    totalActiveDeletedDocs 0
   [junit4]   2> ## Max replicas per node: 2
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 6
   [junit4]   2> ## * policiesTest 6
   [junit4]   2> ## - active           6
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - CREATE             1
   [junit4]   2> ## - CREATESHARD        1
   [junit4]   2> ## - autoscaling        1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2>
   [junit4] IGNOR/A 0.00s J1 | TestSimPolicyCloud.testCreateCollectionAddShardUsingPolicy
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 551462 INFO  (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDataProvider
   [junit4]   2> 551462 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 551462 INFO  (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 551462 DEBUG (ScheduledTrigger-2613-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551462 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 551463 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 551463 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 551463 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 551463 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 551464 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 551464 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 551465 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 551465 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 551465 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion -1
   [junit4]   2> 551466 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 551476 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 551480 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 551480 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 551481 DEBUG (ScheduledTrigger-2616-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551503 DEBUG (ScheduledTrigger-2616-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551530 DEBUG (ScheduledTrigger-2616-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551561 DEBUG (ScheduledTrigger-2616-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551591 DEBUG (ScheduledTrigger-2616-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551613 DEBUG (ScheduledTrigger-2616-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551633 DEBUG (ScheduledTrigger-2616-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551663 DEBUG (ScheduledTrigger-2616-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551673 DEBUG (simCloudManagerPool-2615-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection policiesTest, currentVersion=13
   [junit4]   2> 551673 DEBUG (simCloudManagerPool-2615-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=13
   [junit4]   2> 551673 DEBUG (simCloudManagerPool-2615-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 13
   [junit4]   2> 551689 DEBUG (simCloudManagerPool-2615-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=14
   [junit4]   2> 551689 DEBUG (simCloudManagerPool-2615-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 14
   [junit4]   2> 551690 DEBUG (ScheduledTrigger-2616-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551691 DEBUG (simCloudManagerPool-2615-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection policiesTest, currentVersion=15
   [junit4]   2> 551701 DEBUG (simCloudManagerPool-2615-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=15
   [junit4]   2> 551701 DEBUG (simCloudManagerPool-2615-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 15
   [junit4]   2> 551702 DEBUG (simCloudManagerPool-2615-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for policiesTest / shard1 (currentVersion=16): {"core_node10":{
   [junit4]   2>     "core":"policiesTest_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"policiesTest",
   [junit4]   2>     "node_name":"127.0.0.1:10003_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 551717 DEBUG (simCloudManagerPool-2615-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=16
   [junit4]   2> 551717 DEBUG (ScheduledTrigger-2616-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551717 DEBUG (simCloudManagerPool-2615-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 16
   [junit4]   2> 551717 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=17
   [junit4]   2> 551718 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 17
   [junit4]   2> 551738 DEBUG (ScheduledTrigger-2616-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551752 INFO  (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDataProvider
   [junit4]   2> 551752 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=18
   [junit4]   2> 551752 DEBUG (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 18
   [junit4]   2> 551752 INFO  (TEST-TestSimPolicyCloud.testDataProvider-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 5
   [junit4]   2> ## Empty nodes: 3
   [junit4]   2> ## Dead nodes: 0
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * policiesTest
   [junit4]   2> ##    shardsTotal 1
   [junit4]   2> ##    shardsState {active=1}
   [junit4]   2> ##      shardsWithoutLeader 0
   [junit4]   2> ##    totalReplicas 2
   [junit4]   2> ##      activeReplicas 2
   [junit4]   2> ##      inactiveReplicas 0
   [junit4]   2> ##    totalActiveDocs 0
   [junit4]   2> ##      bufferedDocs 0
   [junit4]   2> ##      maxActiveSliceDocs 0
   [junit4]   2> ##      minActiveSliceDocs 0
   [junit4]   2> ##      avgActiveSliceDocs 0
   [junit4]   2> ##    totalInactiveDocs 0
   [junit4]   2> ##      maxInactiveSliceDocs 0
   [junit4]   2> ##      minInactiveSliceDocs 0
   [junit4]   2> ##      avgInactiveSliceDocs NaN
   [junit4]   2> ##    totalActiveBytes 10,240
   [junit4]   2> ##      maxActiveSliceBytes 10,240
   [junit4]   2> ##      minActiveSliceBytes 10,240
   [junit4]   2> ##      avgActiveSliceBytes 10,240
   [junit4]   2> ##    totalInactiveBytes 0
   [junit4]   2> ##      maxInactiveSliceBytes 0
   [junit4]   2> ##      minInactiveSliceBytes 0
   [junit4]   2> ##      avgInactiveSliceBytes NaN
   [junit4]   2> ##    totalActiveDeletedDocs 0
   [junit4]   2> ## Max replicas per node: 1
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 2
   [junit4]   2> ## * policiesTest 2
   [junit4]   2> ## - active           2
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - ADDROLE            1
   [junit4]   2> ## - CREATE             1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2>
   [junit4]   2> 551758 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateCollectionAddReplica
   [junit4]   2> 551758 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 551758 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 551758 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 551759 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 551759 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 551759 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 551759 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 551759 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 551759 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 551759 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 551760 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 551760 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 551761 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 551761 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 551761 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
   [junit4]   2> 551761 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 551761 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 551762 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 551762 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 551762 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551788 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551811 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551831 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551857 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551888 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551915 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551942 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551967 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551970 DEBUG (simCloudManagerPool-2618-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=19
   [junit4]   2> 551970 DEBUG (simCloudManagerPool-2618-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 19
   [junit4]   2> 551970 DEBUG (simCloudManagerPool-2618-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 551971 DEBUG (simCloudManagerPool-2618-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 551974 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 551974 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 551974 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 551977 DEBUG (simCloudManagerPool-2618-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection testCreateCollectionAddReplica, currentVersion=20
   [junit4]   2> 551977 DEBUG (simCloudManagerPool-2618-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=20
   [junit4]   2> 551977 DEBUG (simCloudManagerPool-2618-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 20
   [junit4]   2> 551996 DEBUG (simCloudManagerPool-2618-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=21
   [junit4]   2> 551996 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 551996 DEBUG (simCloudManagerPool-2618-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 21
   [junit4]   2> 551998 DEBUG (simCloudManagerPool-2618-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection testCreateCollectionAddReplica, currentVersion=22
   [junit4]   2> 552012 DEBUG (simCloudManagerPool-2618-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=22
   [junit4]   2> 552012 DEBUG (simCloudManagerPool-2618-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 22
   [junit4]   2> 552013 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=23
   [junit4]   2> 552013 DEBUG (simCloudManagerPool-2618-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCreateCollectionAddReplica / shard1 (currentVersion=23): {"core_node1":{
   [junit4]   2>     "core":"testCreateCollectionAddReplica_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"testCreateCollectionAddReplica",
   [junit4]   2>     "node_name":"127.0.0.1:10004_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 552013 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 23
   [junit4]   2> 552023 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552045 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552068 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552093 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552115 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552138 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552167 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552189 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552215 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552236 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552265 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552294 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552316 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552346 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552368 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552392 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552416 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552436 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552463 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552489 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552512 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552543 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552568 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552588 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552620 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552646 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552666 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552689 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552710 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552730 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552751 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552776 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552796 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552816 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552837 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552866 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552889 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552912 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552937 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552957 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552978 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 552999 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553021 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553042 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553068 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553100 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553120 DEBUG (ScheduledTrigger-2619-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553144 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553172 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553198 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553225 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553245 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553266 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553289 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553311 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553331 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553360 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553381 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553409 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553435 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553457 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553487 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553514 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553539 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553566 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553593 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553618 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553645 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553669 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553699 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553726 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553749 DEBUG (ScheduledTrigger-2619-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553779 DEBUG (ScheduledTrigger-2619-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553801 DEBUG (ScheduledTrigger-2619-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553801 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateCollectionAddReplica
   [junit4]   2> 553801 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=24
   [junit4]   2> 553801 DEBUG (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 24
   [junit4]   2> 553802 INFO  (TEST-TestSimPolicyCloud.testCreateCollectionAddReplica-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 5
   [junit4]   2> ## Empty nodes: 4
   [junit4]   2> ## Dead nodes: 0
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * testCreateCollectionAddReplica
   [junit4]   2> ##    shardsTotal 1
   [junit4]   2> ##    shardsState {active=1}
   [junit4]   2> ##      shardsWithoutLeader 0
   [junit4]   2> ##    totalReplicas 1
   [junit4]   2> ##      activeReplicas 1
   [junit4]   2> ##      inactiveReplicas 0
   [junit4]   2> ##    totalActiveDocs 0
   [junit4]   2> ##      bufferedDocs 0
   [junit4]   2> ##      maxActiveSliceDocs 0
   [junit4]   2> ##      minActiveSliceDocs 0
   [junit4]   2> ##      avgActiveSliceDocs 0
   [junit4]   2> ##    totalInactiveDocs 0
   [junit4]   2> ##      maxInactiveSliceDocs 0
   [junit4]   2> ##      minInactiveSliceDocs 0
   [junit4]   2> ##      avgInactiveSliceDocs NaN
   [junit4]   2> ##    totalActiveBytes 10,240
   [junit4]   2> ##      maxActiveSliceBytes 10,240
   [junit4]   2> ##      minActiveSliceBytes 10,240
   [junit4]   2> ##      avgActiveSliceBytes 10,240
   [junit4]   2> ##    totalInactiveBytes 0
   [junit4]   2> ##      maxInactiveSliceBytes 0
   [junit4]   2> ##      minInactiveSliceBytes 0
   [junit4]   2> ##      avgInactiveSliceBytes NaN
   [junit4]   2> ##    totalActiveDeletedDocs 0
   [junit4]   2> ## Max replicas per node: 1
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 1
   [junit4]   2> ## * testCreateCollectionAddReplica 1
   [junit4]   2> ## - active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - CREATE             1
   [junit4]   2> ## - autoscaling        1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2>
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimPolicyCloud -Dtests.method=testCreateCollectionAddReplica -Dtests.seed=D59B683037746B7A -Dtests.slow=true -Dtests.locale=nn-NO -Dtests.timezone=America/Moncton -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 2.05s J1 | TestSimPolicyCloud.testCreateCollectionAddReplica <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Timeout waiting for collection to become active
   [junit4]    > Live Nodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]    > Last available state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/23)={
   [junit4]    >   "replicationFactor":"1",
   [junit4]    >   "pullReplicas":"0",
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false",
   [junit4]    >   "nrtReplicas":"1",
   [junit4]    >   "tlogReplicas":"0",
   [junit4]    >   "autoCreated":"true",
   [junit4]    >   "policy":"c1",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "replicas":{"core_node1":{
   [junit4]    >           "core":"testCreateCollectionAddReplica_shard1_replica_n1",
   [junit4]    >           "SEARCHER.searcher.maxDoc":0,
   [junit4]    >           "SEARCHER.searcher.deletedDocs":0,
   [junit4]    >           "INDEX.sizeInBytes":10240,
   [junit4]    >           "node_name":"127.0.0.1:10004_solr",
   [junit4]    >           "state":"active",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]    >           "SEARCHER.searcher.numDocs":0}},
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active"}}}
   [junit4]    > at __randomizedtesting.SeedInfo.seed([D59B683037746B7A:55BB0D1E263783DC]:0)
   [junit4]    > at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)
   [junit4]    > at org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica(TestSimPolicyCloud.java:123)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: last ClusterState: znodeVersion: 24
   [junit4]    > live nodes:[127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]    > collections:{testCreateCollectionAddReplica=DocCollection(testCreateCollectionAddReplica//clusterstate.json/23)={
   [junit4]    >   "replicationFactor":"1",
   [junit4]    >   "pullReplicas":"0",
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false",
   [junit4]    >   "nrtReplicas":"1",
   [junit4]    >   "tlogReplicas":"0",
   [junit4]    >   "autoCreated":"true",
   [junit4]    >   "policy":"c1",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "replicas":{"core_node1":{
   [junit4]    >           "core":"testCreateCollectionAddReplica_shard1_replica_n1",
   [junit4]    >           "SEARCHER.searcher.maxDoc":0,
   [junit4]    >           "SEARCHER.searcher.deletedDocs":0,
   [junit4]    >           "INDEX.sizeInBytes":10240,
   [junit4]    >           "node_name":"127.0.0.1:10004_solr",
   [junit4]    >           "state":"active",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]    >           "SEARCHER.searcher.numDocs":0}},
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active"}}}}, last coll state: DocCollection(testCreateCollectionAddReplica//clusterstate.json/23)={
   [junit4]    >   "replicationFactor":"1",
   [junit4]    >   "pullReplicas":"0",
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false",
   [junit4]    >   "nrtReplicas":"1",
   [junit4]    >   "tlogReplicas":"0",
   [junit4]    >   "autoCreated":"true",
   [junit4]    >   "policy":"c1",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "replicas":{"core_node1":{
   [junit4]    >           "core":"testCreateCollectionAddReplica_shard1_replica_n1",
   [junit4]    >           "SEARCHER.searcher.maxDoc":0,
   [junit4]    >           "SEARCHER.searcher.deletedDocs":0,
   [junit4]    >           "INDEX.sizeInBytes":10240,
   [junit4]    >           "node_name":"127.0.0.1:10004_solr",
   [junit4]    >           "state":"active",
   [junit4]    >           "type":"NRT",
   [junit4]    >           "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]    >           "SEARCHER.searcher.numDocs":0}},
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active"}}}
   [junit4]    > at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:112)
   [junit4]    > at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:64)
   [junit4]    > ... 41 more
   [junit4]   2> 553810 INFO  (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDataProviderPerReplicaDetails
   [junit4]   2> 553810 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 553810 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 553810 INFO  (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 553810 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 553810 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 553811 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 553811 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 553811 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 553811 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 553811 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 553811 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 553811 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 553812 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 553812 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 553813 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion -1
   [junit4]   2> 553813 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 553813 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 553813 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 553813 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 553814 DEBUG (ScheduledTrigger-2622-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553835 DEBUG (ScheduledTrigger-2622-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553861 DEBUG (ScheduledTrigger-2622-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553888 DEBUG (ScheduledTrigger-2622-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553914 DEBUG (ScheduledTrigger-2622-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553941 DEBUG (ScheduledTrigger-2622-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553968 DEBUG (ScheduledTrigger-2622-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 553989 DEBUG (ScheduledTrigger-2622-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 554019 DEBUG (ScheduledTrigger-2622-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 554020 DEBUG (simCloudManagerPool-2621-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection perReplicaDataColl, currentVersion=25
   [junit4]   2> 554020 DEBUG (simCloudManagerPool-2621-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=25
   [junit4]   2> 554020 DEBUG (simCloudManagerPool-2621-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 25
   [junit4]   2> 554036 DEBUG (simCloudManagerPool-2621-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=26
   [junit4]   2> 554036 DEBUG (simCloudManagerPool-2621-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 26
   [junit4]   2> 554039 DEBUG (simCloudManagerPool-2621-thread-1) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection perReplicaDataColl, currentVersion=27
   [junit4]   2> 554047 DEBUG (ScheduledTrigger-2622-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 554053 DEBUG (simCloudManagerPool-2621-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=27
   [junit4]   2> 554053 DEBUG (simCloudManagerPool-2621-thread-2) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 27
   [junit4]   2> 554054 DEBUG (simCloudManagerPool-2621-thread-7) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for perReplicaDataColl / shard1 (currentVersion=28): {"core_node1":{
   [junit4]   2>     "core":"perReplicaDataColl_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"perReplicaDataColl",
   [junit4]   2>     "node_name":"127.0.0.1:10003_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 554072 DEBUG (simCloudManagerPool-2621-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=28
   [junit4]   2> 554072 DEBUG (simCloudManagerPool-2621-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 28
   [junit4]   2> 554073 DEBUG (ScheduledTrigger-2622-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 554083 DEBUG (simCloudManagerPool-2621-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=29
   [junit4]   2> 554084 DEBUG (simCloudManagerPool-2621-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 29
   [junit4]   2> 554093 DEBUG (ScheduledTrigger-2622-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 554099 DEBUG (simCloudManagerPool-2621-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=30
   [junit4]   2> 554100 DEBUG (simCloudManagerPool-2621-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 30
   [junit4]   2> 554111 DEBUG (simCloudManagerPool-2621-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=31
   [junit4]   2> 554111 DEBUG (simCloudManagerPool-2621-thread-6) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 31
   [junit4]   2> 554112 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=32
   [junit4]   2> 554113 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 32
   [junit4]   2> 554114 DEBUG (ScheduledTrigger-2622-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 554139 DEBUG (ScheduledTrigger-2622-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 554160 DEBUG (ScheduledTrigger-2622-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 554180 DEBUG (ScheduledTrigger-2622-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 554201 DEBUG (ScheduledTrigger-2622-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 554220 INFO  (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDataProviderPerReplicaDetails
   [junit4]   2> 554220 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=33
   [junit4]   2> 554220 DEBUG (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 33
   [junit4]   2> 554220 INFO  (TEST-TestSimPolicyCloud.testDataProviderPerReplicaDetails-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 5
   [junit4]   2> ## Empty nodes: 5
   [junit4]   2> ## Dead nodes: 0
   [junit4]   2> ## Collections:
   [junit4]   2> ## Max replicas per node: 0
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - CREATE             1
   [junit4]   2> ## - DELETE             1
   [junit4]   2> ######### Autoscaling event counts ###########


[...truncated too long message...]

eateCollection metricsTest, currentVersion=56
   [junit4]   2> 18471 DEBUG (simCloudManagerPool-52-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=56
   [junit4]   2> 18471 DEBUG (simCloudManagerPool-52-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 56
   [junit4]   2> 18483 DEBUG (simCloudManagerPool-52-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=57
   [junit4]   2> 18483 DEBUG (simCloudManagerPool-52-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 57
   [junit4]   2> 18485 DEBUG (simCloudManagerPool-52-thread-3) [    ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection metricsTest, currentVersion=58
   [junit4]   2> 18495 DEBUG (ScheduledTrigger-53-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
   [junit4]   2> 18501 DEBUG (simCloudManagerPool-52-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=58
   [junit4]   2> 18501 DEBUG (simCloudManagerPool-52-thread-4) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 58
   [junit4]   2> 18501 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=59
   [junit4]   2> 18501 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 59
   [junit4]   2> 18501 DEBUG (simCloudManagerPool-52-thread-5) [    ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for metricsTest / shard1 (currentVersion=60): {"core_node1":{
   [junit4]   2>     "core":"metricsTest_shard1_replica_n1",
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"metricsTest",
   [junit4]   2>     "node_name":"127.0.0.1:10007_solr",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "state":"active",
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0}}
   [junit4]   2> 18501 INFO  (TEST-TestSimPolicyCloud.testMetricsTag-seed#[D59B683037746B7A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testMetricsTag
   [junit4]   2> 18502 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states, currentVersion=60
   [junit4]   2> 18502 DEBUG (TEST-TestSimPolicyCloud.testMetricsTag-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 60
   [junit4]   2> 18502 INFO  (TEST-TestSimPolicyCloud.testMetricsTag-seed#[D59B683037746B7A]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes: 5
   [junit4]   2> ## Empty nodes: 4
   [junit4]   2> ## Dead nodes: 0
   [junit4]   2> ## Collections:
   [junit4]   2> ##  * metricsTest
   [junit4]   2> ##    shardsTotal 1
   [junit4]   2> ##    shardsState {active=1}
   [junit4]   2> ##      shardsWithoutLeader 0
   [junit4]   2> ##    totalReplicas 1
   [junit4]   2> ##      activeReplicas 1
   [junit4]   2> ##      inactiveReplicas 0
   [junit4]   2> ##    totalActiveDocs 0
   [junit4]   2> ##      bufferedDocs 0
   [junit4]   2> ##      maxActiveSliceDocs 0
   [junit4]   2> ##      minActiveSliceDocs 0
   [junit4]   2> ##      avgActiveSliceDocs 0
   [junit4]   2> ##    totalInactiveDocs 0
   [junit4]   2> ##      maxInactiveSliceDocs 0
   [junit4]   2> ##      minInactiveSliceDocs 0
   [junit4]   2> ##      avgInactiveSliceDocs NaN
   [junit4]   2> ##    totalActiveBytes 10,240
   [junit4]   2> ##      maxActiveSliceBytes 10,240
   [junit4]   2> ##      minActiveSliceBytes 10,240
   [junit4]   2> ##      avgActiveSliceBytes 10,240
   [junit4]   2> ##    totalInactiveBytes 0
   [junit4]   2> ##      maxInactiveSliceBytes 0
   [junit4]   2> ##      minInactiveSliceBytes 0
   [junit4]   2> ##      avgInactiveSliceBytes NaN
   [junit4]   2> ##    totalActiveDeletedDocs 0
   [junit4]   2> ## Max replicas per node: 1
   [junit4]   2> ## Min replicas per node: 0
   [junit4]   2> ## Total replicas: 1
   [junit4]   2> ## * metricsTest 1
   [junit4]   2> ## - active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ## - CREATE             1
   [junit4]   2> ## - autoscaling        2
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2>
   [junit4] OK      0.25s J0 | TestSimPolicyCloud.testMetricsTag
   [junit4]   2> 18503 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 18503 DEBUG (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 18504 DEBUG (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 18504 DEBUG (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 18505 DEBUG (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
   [junit4]   2> 18506 DEBUG (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 18506 DEBUG (SUITE-TestSimPolicyCloud-seed#[D59B683037746B7A]-worker) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestSimPolicyCloud_D59B683037746B7A-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=497, maxMBSortInHeap=5.834826159718762, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@36977ff3), locale=nn-NO, timezone=America/Moncton
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=3,threads=1,free=38538704,total=56623104
   [junit4]   2> NOTE: All tests run in this JVM: [TestSimPolicyCloud, TestSimPolicyCloud]
   [junit4] Completed [4/5 (1!)] on J0 in 4.39s, 7 tests, 1 failure <<< FAILURES!
   [junit4]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud
   [junit4] OK      0.42s J1 | TestSimPolicyCloud.testCreateCollectionAddShardWithReplicaTypeUsingPolicy
   [junit4] OK      0.37s J1 | TestSimPolicyCloud.testCreateCollectionAddShardUsingPolicy
   [junit4] OK      0.27s J1 | TestSimPolicyCloud.testDataProvider
   [junit4] OK      0.26s J1 | TestSimPolicyCloud.testCreateCollectionAddReplica
   [junit4] OK      0.40s J1 | TestSimPolicyCloud.testDataProviderPerReplicaDetails
   [junit4] OK      0.54s J1 | TestSimPolicyCloud.testCreateCollectionSplitShard
   [junit4] OK      0.25s J1 | TestSimPolicyCloud.testMetricsTag
   [junit4] Completed [5/5 (1!)] on J1 in 2.56s, 7 tests
   [junit4]
   [junit4]
   [junit4] Tests with failures [seed: D59B683037746B7A]:
   [junit4]   - org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica
   [junit4]
   [junit4]
   [junit4] JVM J0:     0.94 ..    19.09 =    18.14s
   [junit4] JVM J1:     0.94 ..    20.32 =    19.37s
   [junit4] Execution time total: 20 seconds
   [junit4] Tests summary: 5 suites, 35 tests, 1 failure

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1567: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1092: There were test failures: 5 suites, 35 tests, 1 failure [seed: D59B683037746B7A]

Total time: 22 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   1/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=52998fa50e60ce9c7f49167b1ab107347c30d8d6, workspace=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX
[WARNINGS] Computing warning deltas based on reference build #4922
Recording test results
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4925 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4925/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseSerialGC

5 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:51633/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:51633/solr
        at __randomizedtesting.SeedInfo.seed([EEDE29EE60EA9EE6:2F2E50424DBA5441]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:51948/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:51948/solr
        at __randomizedtesting.SeedInfo.seed([EEDE29EE60EA9EE6:2F2E50424DBA5441]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:51967/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:51967/solr
        at __randomizedtesting.SeedInfo.seed([EEDE29EE60EA9EE6:2F2E50424DBA5441]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:52259/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:52259/solr
        at __randomizedtesting.SeedInfo.seed([EEDE29EE60EA9EE6:2F2E50424DBA5441]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:58565/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:58565/solr
        at __randomizedtesting.SeedInfo.seed([EEDE29EE60EA9EE6:2F2E50424DBA5441]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
        at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 13992 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_EEDE29EE60EA9EE6-001/init-core-data-001
   [junit4]   2> 2432800 WARN  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=16 numCloses=16
   [junit4]   2> 2432800 INFO  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2432801 INFO  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2432810 INFO  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2432810 INFO  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_EEDE29EE60EA9EE6-001/tempDir-001
   [junit4]   2> 2432810 INFO  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2432811 INFO  (Thread-2089) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2432811 INFO  (Thread-2089) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2432815 ERROR (Thread-2089) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2432913 INFO  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:58562
   [junit4]   2> 2432926 INFO  (zkConnectionManagerCallback-5309-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2432957 INFO  (jetty-launcher-5306-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 9+181
   [junit4]   2> 2432958 INFO  (jetty-launcher-5306-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 9+181
   [junit4]   2> 2432959 INFO  (jetty-launcher-5306-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2432959 INFO  (jetty-launcher-5306-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2432959 INFO  (jetty-launcher-5306-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2432960 INFO  (jetty-launcher-5306-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d6b654f{/solr,null,AVAILABLE}
   [junit4]   2> 2432962 INFO  (jetty-launcher-5306-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2432962 INFO  (jetty-launcher-5306-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2432962 INFO  (jetty-launcher-5306-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2432962 INFO  (jetty-launcher-5306-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@73f0c761{SSL,[ssl, http/1.1]}{127.0.0.1:58565}
   [junit4]   2> 2432962 INFO  (jetty-launcher-5306-thread-2) [    ] o.e.j.s.Server Started @2433072ms
   [junit4]   2> 2432962 INFO  (jetty-launcher-5306-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@482ed7b2{/solr,null,AVAILABLE}
   [junit4]   2> 2432962 INFO  (jetty-launcher-5306-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58565}
   [junit4]   2> 2432962 ERROR (jetty-launcher-5306-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2432962 INFO  (jetty-launcher-5306-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6a53c018{SSL,[ssl, http/1.1]}{127.0.0.1:58566}
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-1) [    ] o.e.j.s.Server Started @2433072ms
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58566}
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2432963 ERROR (jetty-launcher-5306-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-14T16:55:20.292710Z
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2432963 INFO  (jetty-launcher-5306-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-11-14T16:55:20.292950Z
   [junit4]   2> 2432966 INFO  (zkConnectionManagerCallback-5311-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2432966 INFO  (jetty-launcher-5306-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2432969 INFO  (zkConnectionManagerCallback-5313-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2432969 INFO  (jetty-launcher-5306-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2433271 INFO  (jetty-launcher-5306-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58562/solr
   [junit4]   2> 2433276 INFO  (zkConnectionManagerCallback-5317-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2433307 INFO  (zkConnectionManagerCallback-5319-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2433580 INFO  (jetty-launcher-5306-thread-1) [n:127.0.0.1:58566_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58566_solr
   [junit4]   2> 2433582 INFO  (jetty-launcher-5306-thread-1) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer Overseer (id=72110765190283268-127.0.0.1:58566_solr-n_0000000000) starting
   [junit4]   2> 2433593 INFO  (zkConnectionManagerCallback-5326-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2433601 INFO  (jetty-launcher-5306-thread-1) [n:127.0.0.1:58566_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58562/solr ready
   [junit4]   2> 2433602 INFO  (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:58566_solr
   [junit4]   2> 2433622 INFO  (jetty-launcher-5306-thread-1) [n:127.0.0.1:58566_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58566_solr
   [junit4]   2> 2433626 INFO  (zkCallback-5325-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2433627 INFO  (zkCallback-5318-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2433648 INFO  (jetty-launcher-5306-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58562/solr
   [junit4]   2> 2433664 INFO  (jetty-launcher-5306-thread-1) [n:127.0.0.1:58566_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2433665 INFO  (zkConnectionManagerCallback-5330-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2433668 INFO  (zkConnectionManagerCallback-5333-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2433689 INFO  (jetty-launcher-5306-thread-2) [n:127.0.0.1:58565_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2433701 INFO  (jetty-launcher-5306-thread-1) [n:127.0.0.1:58566_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58566.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c75925e
   [junit4]   2> 2433710 INFO  (jetty-launcher-5306-thread-1) [n:127.0.0.1:58566_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58566.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c75925e
   [junit4]   2> 2433710 INFO  (jetty-launcher-5306-thread-1) [n:127.0.0.1:58566_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58566.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c75925e
   [junit4]   2> 2433711 INFO  (jetty-launcher-5306-thread-1) [n:127.0.0.1:58566_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_EEDE29EE60EA9EE6-001/tempDir-001/node1/.
   [junit4]   2> 2433728 INFO  (jetty-launcher-5306-thread-2) [n:127.0.0.1:58565_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2433729 INFO  (jetty-launcher-5306-thread-2) [n:127.0.0.1:58565_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58565_solr
   [junit4]   2> 2433729 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:58565_solr"} current state version: 0
   [junit4]   2> 2433731 INFO  (zkCallback-5325-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2433731 INFO  (zkCallback-5332-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2433732 INFO  (zkCallback-5318-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2433769 INFO  (zkConnectionManagerCallback-5340-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2433770 INFO  (jetty-launcher-5306-thread-2) [n:127.0.0.1:58565_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2433772 INFO  (jetty-launcher-5306-thread-2) [n:127.0.0.1:58565_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58562/solr ready
   [junit4]   2> 2433772 INFO  (jetty-launcher-5306-thread-2) [n:127.0.0.1:58565_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2433793 INFO  (jetty-launcher-5306-thread-2) [n:127.0.0.1:58565_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58565.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c75925e
   [junit4]   2> 2433806 INFO  (jetty-launcher-5306-thread-2) [n:127.0.0.1:58565_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58565.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c75925e
   [junit4]   2> 2433806 INFO  (jetty-launcher-5306-thread-2) [n:127.0.0.1:58565_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58565.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c75925e
   [junit4]   2> 2433807 INFO  (jetty-launcher-5306-thread-2) [n:127.0.0.1:58565_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_EEDE29EE60EA9EE6-001/tempDir-001/node2/.
   [junit4]   2> 2433888 INFO  (zkConnectionManagerCallback-5343-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2433896 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100305bdfa60009, likely client has closed socket
   [junit4]   2> 2433917 INFO  (zkConnectionManagerCallback-5348-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2433919 INFO  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2433920 INFO  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58562/solr ready
   [junit4]   2> 2433939 INFO  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.c.TestTlogReplica Using legacyCloud?: false
   [junit4]   2> 2433985 INFO  (qtp275683838-14447) [n:127.0.0.1:58565_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2433987 INFO  (qtp275683838-14447) [n:127.0.0.1:58565_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 2434029 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testKillTlogReplica
   [junit4]   2> 2434032 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_kill_tlog_replica&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2434035 INFO  (OverseerThreadFactory-5480-thread-1-processing-n:127.0.0.1:58566_solr) [n:127.0.0.1:58566_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection tlog_replica_test_kill_tlog_replica
   [junit4]   2> 2434044 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "maxShardsPerNode":"100",
   [junit4]   2>   "pullReplicas":"0",
   [junit4]   2>   "tlogReplicas":"2",
   [junit4]   2>   "nrtReplicas":"0",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"0",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 2434165 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2434184 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2434193 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:58566/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 2434193 INFO  (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:58566/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2434198 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:58565/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} current state version: 0
   [junit4]   2> 2434198 INFO  (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:58565/solr",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2434453 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr    x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_tlog_replica_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_kill_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2434464 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr    x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_tlog_replica_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_kill_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 2434464 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr    x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2434469 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t2",
   [junit4]   2>   "core_node_name":"core_node4",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:58565/solr",
   [junit4]   2>   "node_name":"127.0.0.1:58565_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2434478 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t1",
   [junit4]   2>   "core_node_name":"core_node3",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:58566/solr",
   [junit4]   2>   "node_name":"127.0.0.1:58566_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2435478 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2435484 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.IndexSchema [tlog_replica_test_kill_tlog_replica_shard1_replica_t2] Schema name=minimal
   [junit4]   2> 2435485 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2435486 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_kill_tlog_replica_shard1_replica_t2' using configuration from collection tlog_replica_test_kill_tlog_replica, trusted=true
   [junit4]   2> 2435488 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58565.solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t2' (registry 'solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c75925e
   [junit4]   2> 2435488 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrCore [[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_EEDE29EE60EA9EE6-001/tempDir-001/node2/tlog_replica_test_kill_tlog_replica_shard1_replica_t2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_EEDE29EE60EA9EE6-001/tempDir-001/node2/./tlog_replica_test_kill_tlog_replica_shard1_replica_t2/data/]
   [junit4]   2> 2435493 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2435498 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.s.IndexSchema [tlog_replica_test_kill_tlog_replica_shard1_replica_t1] Schema name=minimal
   [junit4]   2> 2435500 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2435500 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_kill_tlog_replica_shard1_replica_t1' using configuration from collection tlog_replica_test_kill_tlog_replica, trusted=true
   [junit4]   2> 2435500 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58566.solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1' (registry 'solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c75925e
   [junit4]   2> 2435500 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.SolrCore [[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_EEDE29EE60EA9EE6-001/tempDir-001/node1/tlog_replica_test_kill_tlog_replica_shard1_replica_t1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_EEDE29EE60EA9EE6-001/tempDir-001/node1/./tlog_replica_test_kill_tlog_replica_shard1_replica_t1/data/]
   [junit4]   2> 2435575 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2435575 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2435578 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2435578 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2435579 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@19ca3f83[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] main]
   [junit4]   2> 2435581 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2435582 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2435582 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2435583 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1617129135576973312
   [junit4]   2> 2435584 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2435584 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2435584 INFO  (searcherExecutor-5492-thread-1-processing-n:127.0.0.1:58565_solr x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2 c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t2] Registered new searcher Searcher@19ca3f83[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2435586 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2435586 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2435588 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@573f82e0[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] main]
   [junit4]   2> 2435590 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2435590 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2435591 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2435591 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1617129135585361920
   [junit4]   2> 2435592 INFO  (searcherExecutor-5493-thread-1-processing-n:127.0.0.1:58566_solr x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1 c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t1] Registered new searcher Searcher@573f82e0[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2435594 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tlog_replica_test_kill_tlog_replica/terms/shard1 to Terms{values={core_node4=0}, version=0}
   [junit4]   2> 2435599 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tlog_replica_test_kill_tlog_replica/terms/shard1 to Terms{values={core_node3=0, core_node4=0}, version=1}
   [junit4]   2> 2435604 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2435604 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2435604 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:58565/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/
   [junit4]   2> 2435604 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica"} current state version: 0
   [junit4]   2> 2435605 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_kill_tlog_replica_shard1_replica_t2 url=https://127.0.0.1:58565/solr START replicas=[https://127.0.0.1:58566/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t1/] nUpdates=100
   [junit4]   2> 2435606 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_kill_tlog_replica_shard1_replica_t2 url=https://127.0.0.1:58565/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2435611 INFO  (qtp2080385080-14457) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2435611 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2435611 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2435611 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ZkController tlog_replica_test_kill_tlog_replica_shard1_replica_t2 stopping background replication from leader
   [junit4]   2> 2435619 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:58565/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/ shard1
   [junit4]   2> 2435621 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "base_url":"https://127.0.0.1:58565/solr",
   [junit4]   2>   "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t2",
   [junit4]   2>   "state":"active"} current state version: 0
   [junit4]   2> 2435786 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2435788 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t2",
   [junit4]   2>   "core_node_name":"core_node4",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:58565/solr",
   [junit4]   2>   "node_name":"127.0.0.1:58565_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2435790 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_tlog_replica_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_kill_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1336
   [junit4]   2> 2435890 INFO  (zkCallback-5332-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2436615 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.ZkController tlog_replica_test_kill_tlog_replica_shard1_replica_t1 starting background replication from leader
   [junit4]   2> 2436615 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:01
   [junit4]   2> 2436616 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 2436616 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2436619 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t1",
   [junit4]   2>   "core_node_name":"core_node3",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:58566/solr",
   [junit4]   2>   "node_name":"127.0.0.1:58566_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "force_set_state":"false",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2436620 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_tlog_replica_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_kill_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2156
   [junit4]   2> 2436624 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2436723 INFO  (zkCallback-5332-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2436724 INFO  (zkCallback-5318-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2437267 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Updated masterUrl to https://127.0.0.1:58565/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/
   [junit4]   2> 2437270 INFO  (qtp275683838-14447) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 2437270 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 2437270 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 2437270 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 2437270 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 2437270 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 2437271 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@28da6eb8[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] main]
   [junit4]   2> 2437272 INFO  (searcherExecutor-5493-thread-1) [    ] o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t1] Registered new searcher Searcher@28da6eb8[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2437625 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_kill_tlog_replica&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2} status=0 QTime=3593
   [junit4]   2> 2437635 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2437641 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2437653 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tlog_replica_test_kill_tlog_replica/terms/shard1 to Terms{values={core_node3=1, core_node4=1}, version=2}
   [junit4]   2> 2437658 INFO  (qtp2080385080-14456) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:58565/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/&wt=javabin&version=2}{add=[1 (1617129137744379904)]} 0 1
   [junit4]   2> 2437658 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1 (1617129137744379904)]} 0 9
   [junit4]   2> 2437661 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:58565/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
   [junit4]   2> 2437665 INFO  (qtp275683838-14447) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1617129137760108544,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2437666 INFO  (qtp275683838-14447) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@a473229 commitCommandVersion:1617129137760108544
   [junit4]   2> 2437674 INFO  (qtp275683838-14447) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@1999da96[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] main]
   [junit4]   2> 2437675 INFO  (qtp275683838-14447) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2437677 INFO  (searcherExecutor-5492-thread-1-processing-n:127.0.0.1:58565_solr x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2 c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t2] Registered new searcher Searcher@1999da96[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C1)))}
   [junit4]   2> 2437677 INFO  (qtp275683838-14447) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:58565/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 12
   [junit4]   2> 2437678 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/update params={_stateVer_=tlog_replica_test_kill_tlog_replica:5&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 17
   [junit4]   2> 2437683 INFO  (qtp2080385080-14456) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2437789 INFO  (qtp2080385080-14456) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2437891 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2438003 INFO  (qtp2080385080-14456) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2438045 INFO  (OverseerCollectionConfigSetProcessor-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2438105 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2438209 INFO  (qtp2080385080-14456) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2438277 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 2438277 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 2438277 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Master's version: 1542214524995
   [junit4]   2> 2438277 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 2438277 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 2438278 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 2438278 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&tlogFiles=false&wt=javabin&version=2&command=filelist} status=0 QTime=0
   [junit4]   2> 2438279 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Number of files in latest index in master: 19
   [junit4]   2> 2438280 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(RAMDirectory@52ecd563 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@405d9399)
   [junit4]   2> 2438280 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_BlockTreeOrds_0.pos&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438282 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438284 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneVarGapDocFreqInterval_0.pos&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438285 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Direct_0.dvdd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438286 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438287 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneVarGapDocFreqInterval_0.tib&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438289 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438290 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_BlockTreeOrds_0.tio&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438296 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_BlockTreeOrds_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438301 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438304 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_BlockTreeOrds_0.tipo&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438314 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Asserting_0.dvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438314 INFO  (qtp2080385080-14454) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2438316 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneVarGapDocFreqInterval_0.tiv&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438317 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Asserting_0.dvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438317 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438319 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Direct_0.dvdm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438320 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneVarGapDocFreqInterval_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438321 INFO  (qtp275683838-14445) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438323 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 2438324 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Bytes downloaded: 3163, Bytes skipped downloading: 0
   [junit4]   2> 2438324 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=3163) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@52ecd563 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@405d9399)
   [junit4]   2> 2438328 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 2438328 INFO  (indexFetcher-5503-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@226434ce[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] main]
   [junit4]   2> 2438330 INFO  (searcherExecutor-5493-thread-1) [    ] o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t1] Registered new searcher Searcher@226434ce[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C1)))}
   [junit4]   2> 2438424 INFO  (qtp2080385080-14456) [n:127.0.0.1:58566_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
   [junit4]   2> 2438430 INFO  (qtp275683838-14448) [n:127.0.0.1:58565_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
   [junit4]   2> 2438431 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.c.ChaosMonkey monkey: kill jetty! 58566
   [junit4]   2> 2438432 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@6a53c018{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2438432 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=937088860
   [junit4]   2> 2438432 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2438432 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1c52a165: rootName = solr_58566, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@2b60a422
   [junit4]   2> 2438445 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2438445 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5664c7e5: rootName = solr_58566, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@12770a06
   [junit4]   2> 2438450 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2438450 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5ad7f751: rootName = solr_58566, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2ef872c3
   [junit4]   2> 2438452 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:58566_solr
   [junit4]   2> 2438453 DEBUG (OverseerStateUpdate-72110765190283268-127.0.0.1:58566_solr-n_0000000000) [n:127.0.0.1:58566_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:58566_solr"} current state version: 0
   [junit4]   2> 2438454 INFO  (TEST-TestTlogReplica.testKillTlogReplica-seed#[EEDE29EE60EA9EE6]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 2438455 INFO  (zkCallback-5325-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2438455 INFO  (zkCallback-5332-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2438455 INFO  (zkCallback-5347-thread-1) [    ] o.a.s.c.c.Zk

[...truncated too long message...]

live in ZooKeeper:/live_nodes/127.0.0.1:52260_solr
   [junit4]   2> 305156 INFO  (jetty-closer-266-thread-1) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:52259_solr
   [junit4]   2> 305157 INFO  (jetty-closer-266-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 305158 INFO  (jetty-closer-266-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 305158 INFO  (zkCallback-349-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 305158 INFO  (zkCallback-356-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 305158 INFO  (zkCallback-386-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 305158 INFO  (zkCallback-393-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 305160 INFO  (jetty-closer-266-thread-2) [    ] o.a.s.c.Overseer Overseer (id=72110952123465749-127.0.0.1:52260_solr-n_0000000004) closing
   [junit4]   2> 305162 INFO  (OverseerStateUpdate-72110952123465749-127.0.0.1:52260_solr-n_0000000004) [n:127.0.0.1:52260_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:52260_solr
   [junit4]   2> 305162 WARN  (OverseerAutoScalingTriggerThread-72110952123465749-127.0.0.1:52260_solr-n_0000000004) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 305167 INFO  (jetty-closer-266-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@44177d19{/solr,null,UNAVAILABLE}
   [junit4]   2> 305167 INFO  (jetty-closer-266-thread-1) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 305169 INFO  (jetty-closer-266-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@44ae3252{/solr,null,UNAVAILABLE}
   [junit4]   2> 305169 INFO  (jetty-closer-266-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 305170 ERROR (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 305170 INFO  (SUITE-TestTlogReplica-seed#[EEDE29EE60EA9EE6]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52257 52257
   [junit4]   2> 305206 INFO  (Thread-155) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52257 52257
   [junit4]   2> 305335 WARN  (Thread-155) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 30 /solr/configs/conf
   [junit4]   2> 28 /solr/collections/tlog_replica_test_create_delete/terms/shard1
   [junit4]   2> 27 /solr/collections/tlog_replica_test_create_delete/terms/shard2
   [junit4]   2> 19 /solr/collections/tlog_replica_test_remove_leader/terms/shard1
   [junit4]   2> 18 /solr/collections/tlog_replica_test_recovery/terms/shard1
   [junit4]   2> 16 /solr/collections/tlog_replica_test_kill_leader/terms/shard1
   [junit4]   2> 16 /solr/collections/tlog_replica_test_kill_tlog_replica/terms/shard1
   [junit4]   2> 15 /solr/aliases.json
   [junit4]   2> 15 /solr/collections/tlog_replica_test_basic_leader_election/terms/shard1
   [junit4]   2> 10 /solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/terms/shard1
   [junit4]   2> 10 /solr/collections/tlog_replica_test_delete_by_id/terms/shard1
   [junit4]   2> 10 /solr/collections/tlog_replica_test_only_leader_indexes/terms/shard1
   [junit4]   2> 7 /solr/security.json
   [junit4]   2> 6 /solr/collections/tlog_replica_test_add_remove_tlog_replica/terms/shard2
   [junit4]   2> 5 /solr/collections/tlog_replica_test_add_remove_tlog_replica/terms/shard1
   [junit4]   2> 4 /solr/collections/tlog_replica_test_add_docs/terms/shard1
   [junit4]   2> 3 /solr/collections/tlog_replica_test_real_time_get/terms/shard1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 21 /solr/collections/tlog_replica_test_create_delete/state.json
   [junit4]   2> 15 /solr/clusterprops.json
   [junit4]   2> 15 /solr/clusterstate.json
   [junit4]   2> 13 /solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/state.json
   [junit4]   2> 10 /solr/collections/tlog_replica_test_remove_leader/state.json
   [junit4]   2> 8 /solr/collections/tlog_replica_test_kill_leader/state.json
   [junit4]   2> 8 /solr/collections/tlog_replica_test_recovery/state.json
   [junit4]   2> 8 /solr/collections/tlog_replica_test_basic_leader_election/state.json
   [junit4]   2> 8 /solr/collections/tlog_replica_test_kill_tlog_replica/state.json
   [junit4]   2> 3 /solr/autoscaling.json
   [junit4]   2> 3 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/72110952123465734-core_node13-n_0000000001
   [junit4]   2> 3 /solr/overseer_elect/election/72110952123465734-127.0.0.1:52260_solr-n_0000000000
   [junit4]   2> 3 /solr/collections/tlog_replica_test_add_remove_tlog_replica/state.json
   [junit4]   2> 3 /solr/collections/tlog_replica_test_only_leader_indexes/state.json
   [junit4]   2> 3 /solr/collections/tlog_replica_test_delete_by_id/state.json
   [junit4]   2> 2 /solr/collections/tlog_replica_test_kill_tlog_replica/leader_elect/shard1/election/72110952123465734-core_node4-n_0000000000
   [junit4]   2> 2 /solr/collections/tlog_replica_test_add_docs/state.json
   [junit4]   2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/72110952123465734-core_node15-n_0000000000
   [junit4]   2> 2 /solr/collections/tlog_replica_test_recovery/leader_elect/shard1/election/72110952123465749-core_node4-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/72110952123465749-127.0.0.1:52260_solr-n_0000000004
   [junit4]   2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/72110952123465741-core_node9-n_0000000001
   [junit4]   2> 2 /solr/collections/tlog_replica_test_real_time_get/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 15 /solr/live_nodes
   [junit4]   2> 15 /solr/collections
   [junit4]   2> 3 /solr/overseer/queue
   [junit4]   2> 3 /solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2> 3 /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 3 /solr/overseer/collection-queue-work
   [junit4]   2> 3 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EEDE29EE60EA9EE6-003
   [junit4]   2> Nov 14, 2018 5:44:25 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80): {foo=PostingsFormat(name=LuceneVarGapDocFreqInterval), title_s=Lucene50(blocksize=128), id=BlockTreeOrds(blocksize=128), foo_s=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{_version_=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Direct), inplace_updatable_int=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1750, maxMBSortInHeap=5.684528722669064, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@7f5199ae), locale=fr-DJ, timezone=America/Denver
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 9 (64-bit)/cpus=3,threads=1,free=45714168,total=242020352
   [junit4]   2> NOTE: All tests run in this JVM: [TestTlogReplica, TestTlogReplica, TestTlogReplica]
   [junit4] Completed [5/5 (5!)] on J0 in 93.40s, 13 tests, 1 failure <<< FAILURES!

[...truncated 14 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1567: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1092: There were test failures: 5 suites, 65 tests, 5 failures [seed: EEDE29EE60EA9EE6]

Total time: 5 minutes 9 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro]   5/5 failed: org.apache.solr.cloud.TestTlogReplica
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=95d01c6583b825b6b87591e4f27002c285ea25fb, workspace=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX
[WARNINGS] Computing warning deltas based on reference build #4924
Recording test results
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]