[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 401 - Failure

classic Classic list List threaded Threaded
38 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 421 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/421/

2 tests failed.
FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:42870/ze/p/forceleader_test_collection

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:42870/ze/p/forceleader_test_collection
        at __randomizedtesting.SeedInfo.seed([7A2C2170CC9B1549:9CBB15B0F519EC28]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:294)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 52 more


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
        at __randomizedtesting.SeedInfo.seed([7A2C2170CC9B1549:F2781EAA626778B1]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:195)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:143)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:138)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1008)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:164)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:108)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14143 lines...]
   [junit4] Suite: org.apache.solr.cloud.ForceLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/init-core-data-001
   [junit4]   2> 1186665 WARN  (SUITE-ForceLeaderTest-seed#[7A2C2170CC9B1549]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 1186665 INFO  (SUITE-ForceLeaderTest-seed#[7A2C2170CC9B1549]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1186668 INFO  (SUITE-ForceLeaderTest-seed#[7A2C2170CC9B1549]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 1186668 INFO  (SUITE-ForceLeaderTest-seed#[7A2C2170CC9B1549]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1186668 INFO  (SUITE-ForceLeaderTest-seed#[7A2C2170CC9B1549]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /ze/p
   [junit4] IGNOR/A 0.00s J2 | ForceLeaderTest.testReplicasInLowerTerms
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 1186685 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1186685 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1186685 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1186785 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer start zk server on port:39115
   [junit4]   2> 1186786 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:39115
   [junit4]   2> 1186786 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 39115
   [junit4]   2> 1186799 INFO  (zkConnectionManagerCallback-6752-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1186802 INFO  (zkConnectionManagerCallback-6754-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1186804 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1186806 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1186809 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1186812 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1186814 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1186816 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1186817 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1186818 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1186819 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1186822 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1186824 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1186825 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1187030 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1187030 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1187030 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1187030 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1187033 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1187033 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1187034 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1187035 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@399cb73c{/ze/p,null,AVAILABLE}
   [junit4]   2> 1187036 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@a932a8f{HTTP/1.1,[http/1.1]}{127.0.0.1:36109}
   [junit4]   2> 1187036 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.e.j.s.Server Started @1187092ms
   [junit4]   2> 1187036 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/tempDir-001/control/data, replicaType=NRT, hostContext=/ze/p, hostPort=37023, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/control-001/cores}
   [junit4]   2> 1187037 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1187037 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1187037 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1187037 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1187037 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1187037 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-03T03:36:07.345Z
   [junit4]   2> 1187045 INFO  (zkConnectionManagerCallback-6756-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1187046 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1187046 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/control-001/solr.xml
   [junit4]   2> 1187052 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1187052 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1187054 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1187293 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39115/solr
   [junit4]   2> 1187296 INFO  (zkConnectionManagerCallback-6760-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1187299 INFO  (zkConnectionManagerCallback-6762-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1187401 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37023_ze%2Fp
   [junit4]   2> 1187401 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.Overseer Overseer (id=72557066327818244-127.0.0.1:37023_ze%2Fp-n_0000000000) starting
   [junit4]   2> 1187410 INFO  (zkConnectionManagerCallback-6769-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1187413 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39115/solr ready
   [junit4]   2> 1187414 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37023_ze%2Fp
   [junit4]   2> 1187414 INFO  (OverseerStateUpdate-72557066327818244-127.0.0.1:37023_ze%2Fp-n_0000000000) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37023_ze%2Fp
   [junit4]   2> 1187417 INFO  (OverseerStateUpdate-72557066327818244-127.0.0.1:37023_ze%2Fp-n_0000000000) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1187417 INFO  (zkCallback-6768-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1187479 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1187521 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1187537 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1187538 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1187540 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/control-001/cores
   [junit4]   2> 1187573 INFO  (zkConnectionManagerCallback-6775-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1187574 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1187575 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39115/solr ready
   [junit4]   2> 1187576 INFO  (SocketProxy-Acceptor-37023) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=40886,localport=37023], receiveBufferSize:531000
   [junit4]   2> 1187582 INFO  (SocketProxy-Acceptor-37023) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36109,localport=44628], receiveBufferSize=530904
   [junit4]   2> 1187583 INFO  (qtp860517021-22114) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37023_ze%252Fp&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1187592 INFO  (OverseerThreadFactory-6752-thread-1-processing-n:127.0.0.1:37023_ze%2Fp) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1187702 INFO  (SocketProxy-Acceptor-37023) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=40890,localport=37023], receiveBufferSize:531000
   [junit4]   2> 1187705 INFO  (SocketProxy-Acceptor-37023) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36109,localport=44632], receiveBufferSize=530904
   [junit4]   2> 1187706 INFO  (qtp860517021-22116) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1187707 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1187709 INFO  (qtp860517021-22113) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1187714 INFO  (qtp860517021-22115) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1187719 INFO  (qtp860517021-22116) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1187733 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1187733 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1188759 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1188799 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1188992 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1189072 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1189072 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1189074 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1189081 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=5.306640625, floorSegmentMB=2.0361328125, forceMergeDeletesPctAllowed=27.10008481451904, segmentsPerTier=23.0, maxCFSSegmentSizeMB=1.1376953125, noCFSRatio=0.84559769647426, deletesPctAllowed=22.204288423066767
   [junit4]   2> 1189090 WARN  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1189165 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1189165 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1189168 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1189168 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1189170 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=4.888671875, floorSegmentMB=2.09375, forceMergeDeletesPctAllowed=17.094270512380866, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.43639855154374063, deletesPctAllowed=42.09031271775156
   [junit4]   2> 1189171 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@109bc40[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1189173 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1189174 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1189175 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1189175 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1621608701082206208
   [junit4]   2> 1189180 INFO  (searcherExecutor-6757-thread-1-processing-n:127.0.0.1:37023_ze%2Fp x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@109bc40[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1189191 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 1189191 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 1189195 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1189195 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1189195 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37023/ze/p/control_collection_shard1_replica_n1/
   [junit4]   2> 1189196 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1189196 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:37023/ze/p/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1189196 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1189197 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/72557066327818244-core_node2-n_0000000000
   [junit4]   2> 1189199 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37023/ze/p/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1189301 INFO  (zkCallback-6761-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1189302 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1189308 INFO  (zkCallback-6761-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1189308 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1576
   [junit4]   2> 1189313 INFO  (qtp860517021-22114) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 1189412 INFO  (zkCallback-6761-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1189412 INFO  (zkCallback-6761-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1189413 INFO  (qtp860517021-22114) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37023_ze%252Fp&wt=javabin&version=2} status=0 QTime=1830
   [junit4]   2> 1189425 INFO  (zkCallback-6761-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1189449 INFO  (zkConnectionManagerCallback-6780-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1189451 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1189452 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39115/solr ready
   [junit4]   2> 1189452 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1189453 INFO  (SocketProxy-Acceptor-37023) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=40900,localport=37023], receiveBufferSize:531000
   [junit4]   2> 1189454 INFO  (SocketProxy-Acceptor-37023) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36109,localport=44642], receiveBufferSize=530904
   [junit4]   2> 1189477 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1189481 INFO  (OverseerCollectionConfigSetProcessor-72557066327818244-127.0.0.1:37023_ze%2Fp-n_0000000000) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1189485 INFO  (OverseerThreadFactory-6752-thread-2-processing-n:127.0.0.1:37023_ze%2Fp) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 1189693 WARN  (OverseerThreadFactory-6752-thread-2-processing-n:127.0.0.1:37023_ze%2Fp) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 1189697 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 1189702 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=225
   [junit4]   2> 1189704 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=3
   [junit4]   2> 1189918 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-1-001 of type NRT
   [junit4]   2> 1189921 WARN  (closeThreadPool-6781-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1189921 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1189921 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1189921 INFO  (closeThreadPool-6781-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1189938 INFO  (closeThreadPool-6781-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1189938 INFO  (closeThreadPool-6781-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1189938 INFO  (closeThreadPool-6781-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1189938 INFO  (closeThreadPool-6781-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@51637695{/ze/p,null,AVAILABLE}
   [junit4]   2> 1189939 INFO  (closeThreadPool-6781-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@73ba7eb{HTTP/1.1,[http/1.1]}{127.0.0.1:36350}
   [junit4]   2> 1189939 INFO  (closeThreadPool-6781-thread-1) [    ] o.e.j.s.Server Started @1189995ms
   [junit4]   2> 1189939 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/ze/p, hostPort=37384, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-1-001/cores}
   [junit4]   2> 1189940 ERROR (closeThreadPool-6781-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1189940 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1189940 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1189940 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1189940 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1189940 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-03T03:36:10.248Z
   [junit4]   2> 1189955 INFO  (zkConnectionManagerCallback-6783-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1189964 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1189964 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-1-001/solr.xml
   [junit4]   2> 1189984 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1189984 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1189999 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1190179 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-2-001 of type NRT
   [junit4]   2> 1190207 WARN  (closeThreadPool-6781-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1190207 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1190207 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1190208 INFO  (closeThreadPool-6781-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1190279 INFO  (closeThreadPool-6781-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1190279 INFO  (closeThreadPool-6781-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1190280 INFO  (closeThreadPool-6781-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1190280 INFO  (closeThreadPool-6781-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@782bfd99{/ze/p,null,AVAILABLE}
   [junit4]   2> 1190282 INFO  (closeThreadPool-6781-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7f38c2ff{HTTP/1.1,[http/1.1]}{127.0.0.1:33096}
   [junit4]   2> 1190282 INFO  (closeThreadPool-6781-thread-2) [    ] o.e.j.s.Server Started @1190338ms
   [junit4]   2> 1190282 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/ze/p, hostPort=46810, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-2-001/cores}
   [junit4]   2> 1190282 ERROR (closeThreadPool-6781-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1190282 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1190283 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1190283 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1190283 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1190283 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-03T03:36:10.591Z
   [junit4]   2> 1190300 INFO  (zkConnectionManagerCallback-6785-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190305 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1190305 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-2-001/solr.xml
   [junit4]   2> 1190310 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1190310 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1190312 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1190467 INFO  (closeThreadPool-6781-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39115/solr
   [junit4]   2> 1190488 INFO  (closeThreadPool-6781-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39115/solr
   [junit4]   2> 1190490 INFO  (zkConnectionManagerCallback-6789-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190493 INFO  (zkConnectionManagerCallback-6793-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190497 INFO  (zkConnectionManagerCallback-6795-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190502 INFO  (zkConnectionManagerCallback-6797-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190508 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1190517 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.c.ZkController Publish node=127.0.0.1:46810_ze%2Fp as DOWN
   [junit4]   2> 1190518 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1190518 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46810_ze%2Fp
   [junit4]   2> 1190520 INFO  (zkCallback-6779-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1190520 INFO  (zkCallback-6768-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1190520 INFO  (zkCallback-6761-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1190521 INFO  (zkCallback-6794-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1190535 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1190540 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[7A2C2170CC9B1549]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-3-001 of type NRT
   [junit4]   2> 1190541 WARN  (closeThreadPool-6781-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1190541 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1190541 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1190542 INFO  (closeThreadPool-6781-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1190543 INFO  (closeThreadPool-6781-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1190543 INFO  (closeThreadPool-6781-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1190543 INFO  (closeThreadPool-6781-thread-3) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1190544 INFO  (closeThreadPool-6781-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@727ffd4f{/ze/p,null,AVAILABLE}
   [junit4]   2> 1190545 INFO  (closeThreadPool-6781-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1f8ab75d{HTTP/1.1,[http/1.1]}{127.0.0.1:38785}
   [junit4]   2> 1190545 INFO  (closeThreadPool-6781-thread-3) [    ] o.e.j.s.Server Started @1190600ms
   [junit4]   2> 1190545 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/ze/p, hostPort=42870, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-3-001/cores}
   [junit4]   2> 1190545 ERROR (closeThreadPool-6781-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1190545 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1190545 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1190545 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1190545 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1190545 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-03T03:36:10.853Z
   [junit4]   2> 1190546 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.c.ZkController Publish node=127.0.0.1:37384_ze%2Fp as DOWN
   [junit4]   2> 1190548 INFO  (zkConnectionManagerCallback-6803-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190552 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1190552 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37384_ze%2Fp
   [junit4]   2> 1190552 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1190553 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-3-001/solr.xml
   [junit4]   2> 1190557 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1190557 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1190559 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1190565 INFO  (zkCallback-6768-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1190565 INFO  (zkCallback-6779-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1190565 INFO  (zkCallback-6761-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1190568 INFO  (zkCallback-6794-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1190573 INFO  (zkCallback-6796-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1190588 INFO  (zkConnectionManagerCallback-6808-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190599 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1190601 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39115/solr ready
   [junit4]   2> 1190602 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1190607 INFO  (zkConnectionManagerCallback-6814-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190609 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1190612 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39115/solr ready
   [junit4]   2> 1190614 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1190643 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1190657 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1190690 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1190691 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1190699 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1190700 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1190702 INFO  (closeThreadPool-6781-thread-1) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-1-001/cores
   [junit4]   2> 1190705 INFO  (closeThreadPool-6781-thread-2) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-2-001/cores
   [junit4]   2> 1190793 INFO  (closeThreadPool-6781-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39115/solr
   [junit4]   2> 1190794 INFO  (zkConnectionManagerCallback-6819-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190804 INFO  (zkConnectionManagerCallback-6821-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190811 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1190818 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.c.ZkController Publish node=127.0.0.1:42870_ze%2Fp as DOWN
   [junit4]   2> 1190819 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1190819 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42870_ze%2Fp
   [junit4]   2> 1190821 INFO  (zkCallback-6768-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1190821 INFO  (zkCallback-6794-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1190821 INFO  (zkCallback-6779-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1190821 INFO  (zkCallback-6761-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1190856 INFO  (zkCallback-6796-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1190858 INFO  (zkCallback-6820-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1190859 INFO  (zkCallback-6807-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1190859 INFO  (zkCallback-6813-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1190886 INFO  (zkConnectionManagerCallback-6828-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1190887 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1190888 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39115/solr ready
   [junit4]   2> 1190889 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1190915 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1190931 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1190931 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1190934 INFO  (closeThreadPool-6781-thread-3) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-3-001/cores
   [junit4]   2> 1191006 INFO  (qtp860517021-22114) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:37384_ze%252Fp&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1191006 INFO  (SocketProxy-Acceptor-37023) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=40930,localport=37023], receiveBufferSize:531000
   [junit4]   2> 1191017 INFO  (SocketProxy-Acceptor-37023) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36109,localport=44672], receiveBufferSize=530904
   [junit4]   2> 1191017 INFO  (SocketProxy-Acceptor-37023) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=40934,localport=37023], receiveBufferSize:531000
   [junit4]   2> 1191026 INFO  (qtp860517021-22116) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:42870_ze%252Fp&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1191026 INFO  (SocketProxy-Acceptor-37023) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36109,localport=44676], receiveBufferSize=530904
   [junit4]   2> 1191027 INFO  (OverseerCollectionConfigSetProcessor-72557066327818244-127.0.0.1:37023_ze%2Fp-n_0000000000) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1191030 INFO  (qtp860517021-22117) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:46810_ze%252Fp&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1191037 INFO  (qtp860517021-22113) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=2
   [junit4]   2> 1191038 INFO  (qtp860517021-22115) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1191038 INFO  (qtp860517021-22113) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1191040 INFO  (qtp860517021-22115) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191041 INFO  (qtp860517021-22113) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191043 INFO  (qtp860517021-22115) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191044 INFO  (qtp860517021-22113) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191045 INFO  (qtp860517021-22115) [n:127.0.0.1:37023_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191046 INFO  (SocketProxy-Acceptor-37384) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56824,localport=37384], receiveBufferSize:531000
   [junit4]   2> 1191053 INFO  (SocketProxy-Acceptor-37384) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36350,localport=42496], receiveBufferSize=530904
   [junit4]   2> 1191058 INFO  (qtp217571462-22181) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191060 INFO  (qtp217571462-22182) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191062 INFO  (qtp217571462-22183) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191063 INFO  (qtp217571462-22184) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191067 INFO  (qtp217571462-22180) [n:127.0.0.1:37384_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191068 INFO  (SocketProxy-Acceptor-42870) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=44696,localport=42870], receiveBufferSize:531000
   [junit4]   2> 1191074 INFO  (SocketProxy-Acceptor-42870) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=38785,localport=57576], receiveBufferSize=530904
   [junit4]   2> 1191093 INFO  (qtp1133455592-22228) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191094 INFO  (qtp1133455592-22229) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191095 INFO  (qtp1133455592-22230) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191096 INFO  (qtp1133455592-22231) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191097 INFO  (qtp1133455592-22227) [n:127.0.0.1:42870_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191100 INFO  (SocketProxy-Acceptor-46810) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=43612,localport=46810], receiveBufferSize:531000
   [junit4]   2> 1191101 INFO  (SocketProxy-Acceptor-46810) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=33096,localport=46222], receiveBufferSize=530904
   [junit4]   2> 1191110 INFO  (qtp308476910-22195) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191111 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191112 INFO  (qtp308476910-22197) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191113 INFO  (qtp308476910-22190) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191116 INFO  (qtp308476910-22194) [n:127.0.0.1:46810_ze%2Fp    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1191117 INFO  (OverseerThreadFactory-6752-thread-3-processing-n:127.0.0.1:37023_ze%2Fp) [n:127.0.0.1:37023_ze%2Fp c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:37384_ze%2Fp for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 1191118 INFO  (OverseerThreadFactory-6752-thread-4-processing-n:127.0.0.1:37023_ze%2Fp) [n:127.0.0.1:37023_ze%2Fp c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:46810_ze%2Fp for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 1191119 INFO  (OverseerThreadFactory-6752-thread-4-processing-n:127.0.0.1:37023_ze%2Fp) [n:127.0.0.1:37023_ze%2Fp c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 1191121 INFO  (SocketProxy-Acceptor-46810) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=43616,localport=46810], receiveBufferSize:531000
   [junit4]   2> 1191122 INFO  (SocketProxy-Acceptor-46810) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=33096,localport=46226], receiveBufferSize=530904
   [junit4]   2> 1191134 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp    x:collection1_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n1&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1191137 INFO  (OverseerThreadFactory-6752-thread-3-processing-n:127.0.0.1:37023_ze%2Fp) [n:127.0.0.1:37023_ze%2Fp c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 1191139 INFO  (SocketProxy-Acceptor-37384) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56840,localport=37384], receiveBufferSize:531000
   [junit4]   2> 1191144 INFO  (SocketProxy-Acceptor-37384) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36350,localport=42512], receiveBufferSize=530904
   [junit4]   2> 1191148 INFO  (qtp217571462-22182) [n:127.0.0.1:37384_ze%2Fp    x:collection1_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n2&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1192156 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1192177 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=test
   [junit4]   2> 1192225 INFO  (qtp217571462-22182) [n:127.0.0.1:37384_ze%2Fp c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1192245 INFO  (qtp217571462-22182) [n:127.0.0.1:37384_ze%2Fp c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.s.IndexSchema [collection1_shard2_replica_n2] Schema name=test
   [junit4]   2> 1192309 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1192335 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 1192336 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n1' (registry 'solr.core.collection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@650469a3
   [junit4]   2> 1192336 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [[collection1_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-2-001/cores/collection1_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_7A2C2170CC9B1549-001/shard-2-001/cores/collection1_shard1_replica_n1/data/]
   [junit4]   2> 1192339 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=5.306640625, floorSegmentMB=2.0361328125, forceMergeDeletesPctAllowed=27.10008481451904, segmentsPerTier=23.0, maxCFSSegmentSizeMB=1.1376953125, noCFSRatio=0.84559769647426, deletesPctAllowed=22.204288423066767
   [junit4]   2> 1192342 WARN  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1192402 INFO  (qtp217571462-22182) [n:127.0.0.1:37384_ze%2Fp c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1192422 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1192423 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1192426 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1192426 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1192427 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=4.888671875, floorSegmentMB=2.09375, forceMergeDeletesPctAllowed=17.094270512380866, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.43639855154374063, deletesPctAllowed=42.09031271775156
   [junit4]   2> 1192428 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@516932f3[collection1_shard1_replica_n1] main]
   [junit4]   2> 1192429 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1192430 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1192431 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1192431 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1621608704496369664
   [junit4]   2> 1192435 INFO  (searcherExecutor-6789-thread-1-processing-n:127.0.0.1:46810_ze%2Fp x:collection1_shard1_replica_n1 c:collection1 s:shard1) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@516932f3[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1192438 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 1192439 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
   [junit4]   2> 1192443 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1192443 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1192443 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46810/ze/p/collection1_shard1_replica_n1/
   [junit4]   2> 1192443 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1192443 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:46810/ze/p/collection1_shard1_replica_n1/ has no replicas
   [junit4]   2> 1192443 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1192445 INFO  (qtp308476910-22196) [n:127.0.0.1:46810_ze%2Fp c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/72557066327818252-core_node3-n_0000000000
   [junit4]   2> 1192447 INFO  (qtp217571462-22182) [n:127.0.0.1:37384_ze%2Fp c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collec

[...truncated too long message...]

vy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1665994598
     [copy] Copying 242 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1665994598
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1665994598

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 59ms :: artifacts dl 4ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 297 minutes 34 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 422 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/422/

3 tests failed.
FAILED:  org.apache.lucene.search.join.TestBlockJoin.testMultiChildQueriesOfDiffParentLevels

Error Message:
CheckIndex failed

Stack Trace:
java.lang.RuntimeException: CheckIndex failed
        at __randomizedtesting.SeedInfo.seed([C780B77D6F043E5D:1665A8B18F92C73B]:0)
        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:305)
        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:285)
        at org.apache.lucene.store.BaseDirectoryWrapper.close(BaseDirectoryWrapper.java:45)
        at org.apache.lucene.search.join.TestBlockJoin.testMultiChildQueriesOfDiffParentLevels(TestBlockJoin.java:1475)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testBelowSearchRate

Error Message:
[Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node10]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node8]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{   "first":"belowRate_collection",   "second":"shard1"}], REPLICA=[core_node6]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:45669_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:43571_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:36248_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:42954_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:37146_solr]}}] expected:<7> but was:<8>

Stack Trace:
java.lang.AssertionError: [Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node10]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node8]}}, Op{action=DELETEREPLICA, hints={COLL_SHARD=[{
  "first":"belowRate_collection",
  "second":"shard1"}], REPLICA=[core_node6]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:45669_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:43571_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:36248_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:42954_solr]}}, Op{action=NONE, hints={SRC_NODE=[127.0.0.1:37146_solr]}}] expected:<7> but was:<8>
        at __randomizedtesting.SeedInfo.seed([45DE88798C547F00:2F732E19D4BCFA13]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testBelowSearchRate(SearchRateTriggerIntegrationTest.java:400)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:44448/pj/fb/forceleader_test_collection

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:44448/pj/fb/forceleader_test_collection
        at __randomizedtesting.SeedInfo.seed([45DE88798C547F00:A349BCB9B5D68661]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:294)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 52 more




Build Log:
[...truncated 9240 lines...]
   [junit4] Suite: org.apache.lucene.search.join.TestBlockJoin
   [junit4]   1> CheckIndex failed
   [junit4]   1> 0.00% total deletions; 1425552 documents; 0 deleteions
   [junit4]   1> Segments file=segments_1 numSegments=4 version=7.7.0 id=50vriaoycfcullfypx2n0woa6
   [junit4]   1>   1 of 4: name=_a maxDoc=1177777
   [junit4]   1>     version=7.7.0
   [junit4]   1>     id=50vriaoycfcullfypx2n0woa2
   [junit4]   1>     codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=16384, maxDocsPerChunk=128, blockSize=1024), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=16384, blockSize=1024))
   [junit4]   1>     compound=false
   [junit4]   1>     numFiles=9
   [junit4]   1>     size (MB)=42.929
   [junit4]   1>     diagnostics = {os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, mergeMaxNumSegments=-1, os.arch=amd64, java.runtime.version=1.8.0_191-b12, source=merge, mergeFactor=5, os.version=4.4.0-112-generic, timestamp=1546575546663}
   [junit4]   1>     no deletions
   [junit4]   1>     test: open reader.........FAILED
   [junit4]   1>     WARNING: exorciseIndex() would remove reference to this segment; full exception:
   [junit4]   1> java.lang.OutOfMemoryError: Java heap space
   [junit4]   1> at org.apache.lucene.util.ArrayUtil.growExact(ArrayUtil.java:323)
   [junit4]   1> at org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexReader.<init>(CompressingStoredFieldsIndexReader.java:76)
   [junit4]   1> at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:134)
   [junit4]   1> at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121)
   [junit4]   1> at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:126)
   [junit4]   1> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
   [junit4]   1> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:678)
   [junit4]   1> at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
   [junit4]   1> at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:285)
   [junit4]   1> at org.apache.lucene.store.BaseDirectoryWrapper.close(BaseDirectoryWrapper.java:45)
   [junit4]   1> at org.apache.lucene.search.join.TestBlockJoin.testMultiChildQueriesOfDiffParentLevels(TestBlockJoin.java:1475)
   [junit4]   1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
   [junit4]   1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
   [junit4]   1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
   [junit4]   1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
   [junit4]   1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
   [junit4]   1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
   [junit4]   1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
   [junit4]   1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
   [junit4]   1>
   [junit4]   1>   2 of 4: name=_b maxDoc=94850
   [junit4]   1>     version=7.7.0
   [junit4]   1>     id=50vriaoycfcullfypx2n0woa3
   [junit4]   1>     codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=16384, maxDocsPerChunk=128, blockSize=1024), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=16384, blockSize=1024))
   [junit4]   1>     compound=true
   [junit4]   1>     numFiles=3
   [junit4]   1>     size (MB)=3.476
   [junit4]   1>     diagnostics = {java.runtime.version=1.8.0_191-b12, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, os=Linux, os.arch=amd64, os.version=4.4.0-112-generic, source=flush, timestamp=1546575570329}
   [junit4]   1>     no deletions
   [junit4]   1>     test: open reader.........OK [took 0.435 sec]
   [junit4]   1>     test: check integrity.....OK [took 0.001 sec]
   [junit4]   1>     test: check live docs.....OK [took 0.000 sec]
   [junit4]   1>     test: field infos.........OK [6 fields] [took 0.000 sec]
   [junit4]   1>     test: field norms.........OK [0 fields] [took 0.000 sec]
   [junit4]   1>     test: terms, freq, prox...OK [5982 terms; 94926 terms/docs pairs; 0 tokens] [took 0.047 sec]
   [junit4]   1>     test: stored fields.......OK [96714 total field count; avg 1.0 fields per doc] [took 0.145 sec]
   [junit4]   1>     test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
   [junit4]   1>     test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET] [took 0.000 sec]
   [junit4]   1>     test: points..............OK [1 fields, 94812 points] [took 0.016 sec]
   [junit4]   1>
   [junit4]   1>   3 of 4: name=_c maxDoc=127957
   [junit4]   1>     version=7.7.0
   [junit4]   1>     id=50vriaoycfcullfypx2n0woa4
   [junit4]   1>     codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=16384, maxDocsPerChunk=128, blockSize=1024), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=16384, blockSize=1024))
   [junit4]   1>     compound=true
   [junit4]   1>     numFiles=3
   [junit4]   1>     size (MB)=4.676
   [junit4]   1>     diagnostics = {java.runtime.version=1.8.0_191-b12, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, os=Linux, os.arch=amd64, os.version=4.4.0-112-generic, source=flush, timestamp=1546575571397}
   [junit4]   1>     no deletions
   [junit4]   1>     test: open reader.........OK [took 0.053 sec]
   [junit4]   1>     test: check integrity.....OK [took 0.001 sec]
   [junit4]   1>     test: check live docs.....OK [took 0.000 sec]
   [junit4]   1>     test: field infos.........OK [6 fields] [took 0.000 sec]
   [junit4]   1>     test: field norms.........OK [0 fields] [took 0.000 sec]
   [junit4]   1>     test: terms, freq, prox...OK [6016 terms; 128057 terms/docs pairs; 0 tokens] [took 0.017 sec]
   [junit4]   1>     test: stored fields.......OK [130475 total field count; avg 1.0 fields per doc] [took 0.092 sec]
   [junit4]   1>     test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
   [junit4]   1>     test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET] [took 0.000 sec]
   [junit4]   1>     test: points..............OK [1 fields, 127907 points] [took 0.007 sec]
   [junit4]   1>
   [junit4]   1>   4 of 4: name=_d maxDoc=24968
   [junit4]   1>     version=7.7.0
   [junit4]   1>     id=50vriaoycfcullfypx2n0woa5
   [junit4]   1>     codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=16384, maxDocsPerChunk=128, blockSize=1024), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=16384, blockSize=1024))
   [junit4]   1>     compound=true
   [junit4]   1>     numFiles=3
   [junit4]   1>     size (MB)=0.937
   [junit4]   1>     diagnostics = {java.runtime.version=1.8.0_191-b12, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, os=Linux, os.arch=amd64, os.version=4.4.0-112-generic, source=flush, timestamp=1546575571677}
   [junit4]   1>     no deletions
   [junit4]   1>     test: open reader.........OK [took 0.006 sec]
   [junit4]   1>     test: check integrity.....OK [took 0.000 sec]
   [junit4]   1>     test: check live docs.....OK [took 0.000 sec]
   [junit4]   1>     test: field infos.........OK [6 fields] [took 0.000 sec]
   [junit4]   1>     test: field norms.........OK [0 fields] [took 0.000 sec]
   [junit4]   1>     test: terms, freq, prox...OK [4071 terms; 24988 terms/docs pairs; 0 tokens] [took 0.006 sec]
   [junit4]   1>     test: stored fields.......OK [25458 total field count; avg 1.0 fields per doc] [took 0.007 sec]
   [junit4]   1>     test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec]
   [junit4]   1>     test: docvalues...........OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET] [took 0.000 sec]
   [junit4]   1>     test: points..............OK [1 fields, 24958 points] [took 0.001 sec]
   [junit4]   1>
   [junit4]   1> WARNING: 1 broken segments (containing 1177777 documents) detected
   [junit4]   1> Took 31.590 sec total.
   [junit4]   1>
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBlockJoin -Dtests.method=testMultiChildQueriesOfDiffParentLevels -Dtests.seed=C780B77D6F043E5D -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ar-DZ -Dtests.timezone=US/Arizona -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    143s J2 | TestBlockJoin.testMultiChildQueriesOfDiffParentLevels <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: CheckIndex failed
   [junit4]    > at __randomizedtesting.SeedInfo.seed([C780B77D6F043E5D:1665A8B18F92C73B]:0)
   [junit4]    > at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:305)
   [junit4]    > at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:285)
   [junit4]    > at org.apache.lucene.store.BaseDirectoryWrapper.close(BaseDirectoryWrapper.java:45)
   [junit4]    > at org.apache.lucene.search.join.TestBlockJoin.testMultiChildQueriesOfDiffParentLevels(TestBlockJoin.java:1475)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/join/test/J2/temp/lucene.search.join.TestBlockJoin_C780B77D6F043E5D-001
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=8248, maxDocsPerChunk=1, blockSize=1), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=8248, blockSize=1)), sim=RandomSimilarity(queryNorm=true): {country=BM25(k1=1.2,b=0.75), parent=DFR I(n)2, docType=DFR I(F)L3(800.0), child2=DFR I(ne)L3(800.0), child3=DFR I(F)3(800.0), child0=DFR I(n)LZ(0.3), child1=DFR I(F)1, parent0=DFR I(F)B1, qualification=DFR I(n)2, skill=IB SPL-D3(800.0), childText=DFR I(F)BZ(0.3), parentText=DFR I(ne)3(800.0), child=IB LL-D2}, locale=ar-DZ, timezone=US/Arizona
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=188032256,total=358088704
   [junit4]   2> NOTE: All tests run in this JVM: [TestBlockJoinSorting, TestBlockJoin]
   [junit4] Completed [8/8 (1!)] on J2 in 167.42s, 18 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J2: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/join/test/temp/junit4-J2-20190104_041720_5748905361433971029672.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid4336.hprof ...
   [junit4] Heap dump file created [501508313 bytes in 3.363 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 5302 lines...]
   [junit4] Suite: org.apache.solr.cloud.ForceLeaderTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/init-core-data-001
   [junit4]   2> 2665116 WARN  (SUITE-ForceLeaderTest-seed#[45DE88798C547F00]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 2665116 INFO  (SUITE-ForceLeaderTest-seed#[45DE88798C547F00]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2665117 INFO  (SUITE-ForceLeaderTest-seed#[45DE88798C547F00]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 2665118 INFO  (SUITE-ForceLeaderTest-seed#[45DE88798C547F00]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2665118 INFO  (SUITE-ForceLeaderTest-seed#[45DE88798C547F00]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /pj/fb
   [junit4] IGNOR/A 0.00s J2 | ForceLeaderTest.testReplicasInLowerTerms
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 2665137 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2665142 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2665142 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2665242 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer start zk server on port:42925
   [junit4]   2> 2665242 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:42925
   [junit4]   2> 2665242 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 42925
   [junit4]   2> 2665261 INFO  (zkConnectionManagerCallback-4590-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2665263 INFO  (zkConnectionManagerCallback-4592-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2665269 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2665270 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2665271 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2665272 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2665273 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2665274 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2665275 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2665276 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2665276 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2665277 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2665278 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2665279 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2665452 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2665452 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2665452 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2665452 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2665495 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2665495 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2665495 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2665500 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@342fc7db{/pj/fb,null,AVAILABLE}
   [junit4]   2> 2665500 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@70d8309a{HTTP/1.1,[http/1.1]}{127.0.0.1:37532}
   [junit4]   2> 2665500 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.e.j.s.Server Started @2665543ms
   [junit4]   2> 2665500 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/tempDir-001/control/data, replicaType=NRT, hostContext=/pj/fb, hostPort=44448, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/control-001/cores}
   [junit4]   2> 2665501 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2665501 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2665501 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2665501 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2665501 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2665501 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-04T07:30:20.805Z
   [junit4]   2> 2665503 INFO  (zkConnectionManagerCallback-4594-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2665504 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2665504 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/control-001/solr.xml
   [junit4]   2> 2665507 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2665507 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2665508 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2665771 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42925/solr
   [junit4]   2> 2665803 INFO  (zkConnectionManagerCallback-4598-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2665819 INFO  (zkConnectionManagerCallback-4600-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2665893 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44448_pj%2Ffb
   [junit4]   2> 2665894 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.Overseer Overseer (id=73953184918274052-127.0.0.1:44448_pj%2Ffb-n_0000000000) starting
   [junit4]   2> 2665914 INFO  (zkConnectionManagerCallback-4607-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2665917 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42925/solr ready
   [junit4]   2> 2665927 INFO  (OverseerStateUpdate-73953184918274052-127.0.0.1:44448_pj%2Ffb-n_0000000000) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:44448_pj%2Ffb
   [junit4]   2> 2665938 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44448_pj%2Ffb
   [junit4]   2> 2665939 INFO  (zkCallback-4599-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2665951 INFO  (zkCallback-4606-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2665971 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2665991 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2665999 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2665999 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2666001 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/control-001/cores
   [junit4]   2> 2666127 INFO  (zkConnectionManagerCallback-4613-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2666128 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2666129 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42925/solr ready
   [junit4]   2> 2666130 INFO  (SocketProxy-Acceptor-44448) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=35946,localport=44448], receiveBufferSize:531000
   [junit4]   2> 2666169 INFO  (SocketProxy-Acceptor-44448) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37532,localport=53818], receiveBufferSize=530904
   [junit4]   2> 2666180 INFO  (qtp1071652538-21517) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44448_pj%252Ffb&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2666187 INFO  (OverseerThreadFactory-6019-thread-1-processing-n:127.0.0.1:44448_pj%2Ffb) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2666292 INFO  (SocketProxy-Acceptor-44448) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=35950,localport=44448], receiveBufferSize:531000
   [junit4]   2> 2666293 INFO  (SocketProxy-Acceptor-44448) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37532,localport=53822], receiveBufferSize=530904
   [junit4]   2> 2666294 INFO  (qtp1071652538-21519) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2666295 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2666307 INFO  (qtp1071652538-21516) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2666308 INFO  (qtp1071652538-21518) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2666309 INFO  (qtp1071652538-21519) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2666323 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2666323 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2667338 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 2667366 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2667469 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2667482 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2667483 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2667483 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2667485 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=9, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=57.65234375, floorSegmentMB=0.2880859375, forceMergeDeletesPctAllowed=2.075727760524905, segmentsPerTier=20.0, maxCFSSegmentSizeMB=2.1982421875, noCFSRatio=0.1763358407836699, deletesPctAllowed=43.09784516826911
   [junit4]   2> 2667488 WARN  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2667539 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2667539 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2667541 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2667541 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2667542 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2667544 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3373bb30[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2667545 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2667545 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2667545 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2667546 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1621714034071961600
   [junit4]   2> 2667548 INFO  (searcherExecutor-6024-thread-1-processing-n:127.0.0.1:44448_pj%2Ffb x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@3373bb30[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2667557 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 2667557 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 2667560 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2667560 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2667560 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44448/pj/fb/control_collection_shard1_replica_n1/
   [junit4]   2> 2667561 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2667561 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:44448/pj/fb/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2667561 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2667561 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/73953184918274052-core_node2-n_0000000000
   [junit4]   2> 2667562 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44448/pj/fb/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2667664 INFO  (zkCallback-4599-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2667665 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2667668 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1345
   [junit4]   2> 2667670 INFO  (qtp1071652538-21517) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2667768 INFO  (zkCallback-4599-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2667768 INFO  (zkCallback-4599-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2667768 INFO  (qtp1071652538-21517) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44448_pj%252Ffb&wt=javabin&version=2} status=0 QTime=1588
   [junit4]   2> 2667775 INFO  (zkConnectionManagerCallback-4618-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2667786 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2667787 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42925/solr ready
   [junit4]   2> 2667787 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2667787 INFO  (SocketProxy-Acceptor-44448) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=35956,localport=44448], receiveBufferSize:531000
   [junit4]   2> 2667788 INFO  (SocketProxy-Acceptor-44448) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37532,localport=53828], receiveBufferSize=530904
   [junit4]   2> 2667803 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2667806 INFO  (OverseerCollectionConfigSetProcessor-73953184918274052-127.0.0.1:44448_pj%2Ffb-n_0000000000) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2667806 INFO  (OverseerThreadFactory-6019-thread-2-processing-n:127.0.0.1:44448_pj%2Ffb) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2668010 WARN  (OverseerThreadFactory-6019-thread-2-processing-n:127.0.0.1:44448_pj%2Ffb) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2668020 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2668021 INFO  (qtp1071652538-21520) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=217
   [junit4]   2> 2668029 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=3
   [junit4]   2> 2668191 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-1-001 of type NRT
   [junit4]   2> 2668197 WARN  (closeThreadPool-4619-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2668197 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2668197 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2668197 INFO  (closeThreadPool-4619-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2668210 INFO  (closeThreadPool-4619-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2668210 INFO  (closeThreadPool-4619-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2668210 INFO  (closeThreadPool-4619-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2668211 INFO  (closeThreadPool-4619-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5243123f{/pj/fb,null,AVAILABLE}
   [junit4]   2> 2668211 INFO  (closeThreadPool-4619-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@56208ba3{HTTP/1.1,[http/1.1]}{127.0.0.1:46550}
   [junit4]   2> 2668212 INFO  (closeThreadPool-4619-thread-1) [    ] o.e.j.s.Server Started @2668254ms
   [junit4]   2> 2668212 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/pj/fb, hostPort=41414, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-1-001/cores}
   [junit4]   2> 2668212 ERROR (closeThreadPool-4619-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2668212 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2668212 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2668212 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2668212 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2668212 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-04T07:30:23.516Z
   [junit4]   2> 2668239 INFO  (zkConnectionManagerCallback-4621-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2668247 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2668247 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-1-001/solr.xml
   [junit4]   2> 2668251 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2668251 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2668252 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2668323 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-2-001 of type NRT
   [junit4]   2> 2668339 WARN  (closeThreadPool-4619-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2668339 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2668339 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2668339 INFO  (closeThreadPool-4619-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2668431 INFO  (closeThreadPool-4619-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2668431 INFO  (closeThreadPool-4619-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2668431 INFO  (closeThreadPool-4619-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2668436 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[45DE88798C547F00]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-3-001 of type NRT
   [junit4]   2> 2668439 INFO  (closeThreadPool-4619-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@482ae8ed{/pj/fb,null,AVAILABLE}
   [junit4]   2> 2668439 INFO  (closeThreadPool-4619-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@74c8041e{HTTP/1.1,[http/1.1]}{127.0.0.1:46292}
   [junit4]   2> 2668439 INFO  (closeThreadPool-4619-thread-2) [    ] o.e.j.s.Server Started @2668482ms
   [junit4]   2> 2668439 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/pj/fb, hostPort=43641, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-2-001/cores}
   [junit4]   2> 2668440 ERROR (closeThreadPool-4619-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2668440 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2668440 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2668440 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2668440 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2668440 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-04T07:30:23.744Z
   [junit4]   2> 2668444 WARN  (closeThreadPool-4619-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2668444 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2668444 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2668444 INFO  (closeThreadPool-4619-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2668447 INFO  (closeThreadPool-4619-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2668447 INFO  (closeThreadPool-4619-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2668447 INFO  (closeThreadPool-4619-thread-3) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2668448 INFO  (closeThreadPool-4619-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@32f1c9d3{/pj/fb,null,AVAILABLE}
   [junit4]   2> 2668448 INFO  (closeThreadPool-4619-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@457913b7{HTTP/1.1,[http/1.1]}{127.0.0.1:43032}
   [junit4]   2> 2668448 INFO  (closeThreadPool-4619-thread-3) [    ] o.e.j.s.Server Started @2668490ms
   [junit4]   2> 2668448 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/pj/fb, hostPort=34295, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-3-001/cores}
   [junit4]   2> 2668448 ERROR (closeThreadPool-4619-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2668448 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2668448 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2668448 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2668448 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2668448 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-04T07:30:23.752Z
   [junit4]   2> 2668449 INFO  (zkConnectionManagerCallback-4623-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2668451 INFO  (zkConnectionManagerCallback-4625-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2668451 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2668451 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-2-001/solr.xml
   [junit4]   2> 2668455 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2668455 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2668456 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2668456 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2668456 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-3-001/solr.xml
   [junit4]   2> 2668458 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2668458 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2668460 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2668460 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x106bc07a510000a, likely client has closed socket
   [junit4]   2> 2668515 INFO  (closeThreadPool-4619-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42925/solr
   [junit4]   2> 2668539 INFO  (zkConnectionManagerCallback-4629-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2668579 INFO  (zkConnectionManagerCallback-4631-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2668582 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2668584 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.c.ZkController Publish node=127.0.0.1:41414_pj%2Ffb as DOWN
   [junit4]   2> 2668585 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2668585 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41414_pj%2Ffb
   [junit4]   2> 2668586 INFO  (zkCallback-4599-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2668586 INFO  (zkCallback-4606-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2668602 INFO  (zkCallback-4617-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2668612 INFO  (closeThreadPool-4619-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42925/solr
   [junit4]   2> 2668613 INFO  (zkConnectionManagerCallback-4637-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2668614 INFO  (zkCallback-4630-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2668632 INFO  (zkConnectionManagerCallback-4639-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2668632 INFO  (zkConnectionManagerCallback-4644-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2668633 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2668634 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42925/solr ready
   [junit4]   2> 2668635 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2668639 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2668641 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.c.ZkController Publish node=127.0.0.1:43641_pj%2Ffb as DOWN
   [junit4]   2> 2668642 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2668642 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43641_pj%2Ffb
   [junit4]   2> 2669111 INFO  (zkCallback-4606-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669111 INFO  (zkCallback-4599-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669127 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2669135 INFO  (zkCallback-4630-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669146 INFO  (zkCallback-4643-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669147 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2669147 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2669148 INFO  (closeThreadPool-4619-thread-1) [n:127.0.0.1:41414_pj%2Ffb    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-1-001/cores
   [junit4]   2> 2669151 INFO  (zkCallback-4617-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669153 INFO  (zkCallback-4638-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2669171 INFO  (zkConnectionManagerCallback-4652-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2669172 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2669172 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42925/solr ready
   [junit4]   2> 2669173 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2669188 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2669202 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2669202 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2669203 INFO  (closeThreadPool-4619-thread-2) [n:127.0.0.1:43641_pj%2Ffb    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-2-001/cores
   [junit4]   2> 2669355 INFO  (closeThreadPool-4619-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42925/solr
   [junit4]   2> 2669357 INFO  (zkConnectionManagerCallback-4657-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2669364 INFO  (zkConnectionManagerCallback-4659-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2669367 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2669371 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.c.ZkController Publish node=127.0.0.1:34295_pj%2Ffb as DOWN
   [junit4]   2> 2669372 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2669372 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34295_pj%2Ffb
   [junit4]   2> 2669373 INFO  (zkCallback-4617-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2669373 INFO  (zkCallback-4630-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2669373 INFO  (zkCallback-4651-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2669373 INFO  (zkCallback-4599-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2669373 INFO  (zkCallback-4638-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2669373 INFO  (zkCallback-4643-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2669374 INFO  (zkCallback-4606-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2669374 INFO  (zkCallback-4658-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2669391 INFO  (zkConnectionManagerCallback-4666-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2669392 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2669393 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42925/solr ready
   [junit4]   2> 2669393 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2669408 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2669416 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2669416 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@136227c5
   [junit4]   2> 2669417 INFO  (closeThreadPool-4619-thread-3) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_45DE88798C547F00-001/shard-3-001/cores
   [junit4]   2> 2669458 INFO  (SocketProxy-Acceptor-34295) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57238,localport=34295], receiveBufferSize:531000
   [junit4]   2> 2669458 INFO  (SocketProxy-Acceptor-34295) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43032,localport=34454], receiveBufferSize=530904
   [junit4]   2> 2669459 INFO  (SocketProxy-Acceptor-34295) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57242,localport=34295], receiveBufferSize:531000
   [junit4]   2> 2669464 INFO  (SocketProxy-Acceptor-34295) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43032,localport=34458], receiveBufferSize=530904
   [junit4]   2> 2669464 INFO  (qtp1269166789-21611) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:43641_pj%252Ffb&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2669466 INFO  (OverseerCollectionConfigSetProcessor-73953184918274052-127.0.0.1:44448_pj%2Ffb-n_0000000000) [n:127.0.0.1:44448_pj%2Ffb    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2669468 INFO  (qtp1269166789-21612) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34295_pj%252Ffb&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2669468 INFO  (SocketProxy-Acceptor-34295) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57246,localport=34295], receiveBufferSize:531000
   [junit4]   2> 2669469 INFO  (SocketProxy-Acceptor-34295) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43032,localport=34462], receiveBufferSize=530904
   [junit4]   2> 2669476 INFO  (qtp1269166789-21609) [n:127.0.0.1:34295_pj%2Ffb    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:41414_pj%252Ffb&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2669492 INFO  (SocketProxy-Acceptor-34295) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57250,localport=34295], receiveBufferSize:531000
   [junit4]   2> 2669492 INFO  (Socke

[...truncated too long message...]

j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7faed518{/solr,null,UNAVAILABLE}
   [junit4]   2> 5987683 INFO  (jetty-closer-75601-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 5987683 INFO  (zkCallback-75570-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 5987686 INFO  (jetty-closer-75601-thread-5) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 5987686 INFO  (jetty-closer-75601-thread-5) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@774f4e4b: rootName = solr_42954, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@e155bb2
   [junit4]   2> 5987688 INFO  (jetty-closer-75601-thread-4) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@2c8e4686{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 5987688 INFO  (jetty-closer-75601-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@79994528{/solr,null,UNAVAILABLE}
   [junit4]   2> 5987688 INFO  (jetty-closer-75601-thread-4) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 5987689 INFO  (jetty-closer-75601-thread-5) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 5987689 INFO  (jetty-closer-75601-thread-5) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@137d10f1: rootName = solr_42954, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5a64f167
   [junit4]   2> 5987689 INFO  (jetty-closer-75601-thread-5) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 5987705 INFO  (zkCallback-75578-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 5987707 INFO  (jetty-closer-75601-thread-5) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1987f9b1{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 5987708 INFO  (jetty-closer-75601-thread-5) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@66d54965{/solr,null,UNAVAILABLE}
   [junit4]   2> 5987708 INFO  (jetty-closer-75601-thread-5) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 5987711 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[45DE88798C547F00]-worker) [    ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
   [junit4]   2> 5987720 WARN  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 52 /solr/collections/aboveRate_collection/terms/shard1
   [junit4]   2> 37 /solr/collections/belowRate_collection/terms/shard1
   [junit4]   2> 13 /solr/clusterprops.json
   [junit4]   2> 13 /solr/aliases.json
   [junit4]   2> 10 /solr/configs/conf
   [junit4]   2> 5 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 91 /solr/collections/belowRate_collection/state.json
   [junit4]   2> 64 /solr/collections/aboveRate_collection/state.json
   [junit4]   2> 13 /solr/clusterstate.json
   [junit4]   2> 3 /solr/aliases.json
   [junit4]   2> 2 /solr/overseer_elect/election/73953397000568845-127.0.0.1:43571_solr-n_0000000001
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 14 /solr/collections
   [junit4]   2> 13 /solr/live_nodes
   [junit4]   2>
   [junit4]   2> 5987723 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[45DE88798C547F00]-worker) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:38748
   [junit4]   2> 5987723 INFO  (SUITE-SearchRateTriggerIntegrationTest-seed#[45DE88798C547F00]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 38748
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.SearchRateTriggerIntegrationTest_45DE88798C547F00-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=hi, timezone=Kwajalein
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=301154112,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [V2ApiIntegrationTest, ResponseLogComponentTest, TestWriterPerf, CdcrBootstrapTest, HttpSolrCallGetCoreTest, TestSSLRandomization, ImplicitSnitchTest, TestWordDelimiterFilterFactory, ByteBuffersDirectoryFactoryTest, TestRealTimeGet, NodeMarkersRegistrationTest, TestPushWriter, AtomicUpdatesTest, TriLevelCompositeIdRoutingTest, TestFacetMethods, MoveReplicaHDFSTest, DistributedQueryElevationComponentTest, TestReload, TestAuthenticationFramework, TestBinaryField, TestDFRSimilarityFactory, TestCodecSupport, TestSuggestSpellingConverter, TestPayloadCheckQParserPlugin, HDFSCollectionsAPITest, TestReplicationHandler, TestSolrDeletionPolicy2, TestMiniSolrCloudClusterSSL, DistanceUnitsTest, BinaryUpdateRequestHandlerTest, UUIDUpdateProcessorFallbackTest, SolrCloudReportersTest, TestCustomSort, HdfsDirectoryFactoryTest, TestTolerantUpdateProcessorCloud, TestCharFilters, TestLMJelinekMercerSimilarityFactory, SpatialFilterTest, TestSchemaNameResource, DateFieldTest, TestPullReplica, OverseerTest, TestComplexPhraseLeadingWildcard, BadCopyFieldTest, HdfsAutoAddReplicasIntegrationTest, TestDefaultStatsCache, TestJsonFacets, DistributedFacetExistsSmallTest, CollectionsAPISolrJTest, TestInitParams, PreAnalyzedFieldManagedSchemaCloudTest, TestSimDistribStateManager, TestSolrCloudWithKerberosAlt, PhrasesIdentificationComponentTest, TestFieldResource, PreAnalyzedUpdateProcessorTest, TestMultiWordSynonyms, OverriddenZkACLAndCredentialsProvidersTest, TestDistribDocBasedVersion, HdfsUnloadDistributedZkTest, DistributedTermsComponentTest, DistributedIntervalFacetingTest, TestValueSourceCache, ForceLeaderTest, DistributedFacetPivotSmallAdvancedTest, MigrateRouteKeyTest, TestTrie, SignatureUpdateProcessorFactoryTest, TestManagedSchemaThreadSafety, TestSubQueryTransformerDistrib, TestNestedDocsSort, TaggingAttributeTest, SearchHandlerTest, ClusterStateUpdateTest, ReturnFieldsTest, HdfsLockFactoryTest, CoreAdminRequestStatusTest, TestManagedStopFilterFactory, DistributedFacetSimpleRefinementLongTailTest, ExternalFileFieldSortTest, ZookeeperStatusHandlerTest, AutoScalingHandlerTest, TestRestoreCore, HighlighterConfigTest, TestRequestStatusCollectionAPI, DeleteReplicaTest, UninvertDocValuesMergePolicyTest, OutputWriterTest, TestExportWriter, SolrJmxReporterCloudTest, SimpleCollectionCreateDeleteTest, TestAtomicUpdateErrorCases, AlternateDirectoryTest, OrderedExecutorTest, TestExactStatsCache, TestSimTriggerIntegration, TestDynamicFieldCollectionResource, SuggestComponentTest, TestSimpleTrackingShardHandler, SpellCheckCollatorWithCollapseTest, TestAnalyzeInfixSuggestions, FileBasedSpellCheckerTest, CdcrRequestHandlerTest, TimeRoutedAliasUpdateProcessorTest, TestUtils, TestCloudJSONFacetJoinDomain, TestPerFieldSimilarityWithDefaultOverride, HttpPartitionTest, TestExtendedDismaxParser, TestJettySolrRunner, DistributedQueueTest, DistributedMLTComponentTest, TestLegacyNumericUtils, HdfsSyncSliceTest, TestFileDictionaryLookup, ZkNodePropsTest, TestFieldCache, ManagedSchemaRoundRobinCloudTest, NumericFieldsTest, TestJsonRequest, TestCustomStream, TestLegacyNumericRangeQueryBuilder, BasicAuthStandaloneTest, TestQueryTypes, TestGraphMLResponseWriter, SolrMetricsIntegrationTest, VersionInfoTest, TestXIncludeConfig, TestJoin, SliceStateTest, TestDynamicURP, TestHttpShardHandlerFactory, TestSimExtremeIndexing, CoreSorterTest, SolrXmlInZkTest, CdcrUpdateLogTest, TestEmbeddedSolrServerAdminHandler, TestDocumentBuilder, TestGroupingSearch, TestLuceneMatchVersion, TokenizerChainTest, ConnectionReuseTest, ActionThrottleTest, AddReplicaTest, AssignBackwardCompatibilityTest, AsyncCallRequestStatusResponseTest, BasicDistributedZk2Test, BasicDistributedZkTest, BasicZkTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, ChaosMonkeySafeLeaderTest, CleanupOldIndexTest, CollectionPropsTest, ConnectionManagerTest, DeleteInactiveReplicaTest, DistribCursorPagingTest, LeaderElectionContextKeyTest, LeaderElectionTest, LeaderFailureAfterFreshStartTest, OverseerModifyCollectionTest, RecoveryAfterSoftCommitTest, ReplaceNodeNoTargetTest, ReplaceNodeTest, TestCloudInspectUtil, TestCloudPhrasesIdentificationComponent, TestCloudPseudoReturnFields, TestCloudRecovery, TestCloudSearcherWarming, TestDistributedMap, TestDownShardTolerantSearch, TestExclusionRuleCollectionAccess, TestLeaderElectionWithEmptyReplica, TestOnReconnectListenerSupport, TestRandomFlRTGCloud, TestSkipOverseerOperations, TestStressInPlaceUpdates, TestStressLiveNodes, CollectionReloadTest, ConcurrentDeleteAndCreateCollectionTest, ComputePlanActionTest, NodeAddedTriggerTest, ScheduledTriggerIntegrationTest, ScheduledTriggerTest, SearchRateTriggerIntegrationTest]
   [junit4] Completed [545/835 (2!)] on J2 in 86.73s, 3 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 9863 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:651: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid4336.hprof

Total time: 592 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 423 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/423/

2 tests failed.
FAILED:  org.apache.solr.cloud.LIROnShardRestartTest.testAllReplicasInLIR

Error Message:
Path must not end with / character

Stack Trace:
java.lang.IllegalArgumentException: Path must not end with / character
        at __randomizedtesting.SeedInfo.seed([E36FE4B4C0CCB6C9:B9F7DE72BE4CD12E]:0)
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:58)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1523)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346)
        at org.apache.solr.cloud.LIROnShardRestartTest.testAllReplicasInLIR(LIROnShardRestartTest.java:168)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.LIROnShardRestartTest.testSeveralReplicasInLIR

Error Message:
Error from server at https://127.0.0.1:44062/solr: Cannot create collection severalReplicasInLIR. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 2. This allows a maximum of 2 to be created. Value of numShards is 1, value of nrtReplicas is 3, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 3 shards to be created (higher than the allowed number)

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:44062/solr: Cannot create collection severalReplicasInLIR. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 2. This allows a maximum of 2 to be created. Value of numShards is 1, value of nrtReplicas is 3, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 3 shards to be created (higher than the allowed number)
        at __randomizedtesting.SeedInfo.seed([E36FE4B4C0CCB6C9:65357380E096FB90]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.LIROnShardRestartTest.testSeveralReplicasInLIR(LIROnShardRestartTest.java:190)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14680 lines...]
   [junit4] Suite: org.apache.solr.cloud.LIROnShardRestartTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/init-core-data-001
   [junit4]   2> 2747882 WARN  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=22 numCloses=22
   [junit4]   2> 2747882 INFO  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2747884 INFO  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2747885 INFO  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2747886 INFO  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001
   [junit4]   2> 2747886 INFO  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2747887 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2747887 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2747987 INFO  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:43497
   [junit4]   2> 2747987 INFO  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43497
   [junit4]   2> 2747987 INFO  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 43497
   [junit4]   2> 2747992 INFO  (zkConnectionManagerCallback-64418-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2747998 INFO  (zkConnectionManagerCallback-64420-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2747999 INFO  (zkConnectionManagerCallback-64422-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748003 WARN  (jetty-launcher-64423-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2748003 INFO  (jetty-launcher-64423-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2748003 INFO  (jetty-launcher-64423-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2748004 INFO  (jetty-launcher-64423-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2748004 WARN  (jetty-launcher-64423-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2748004 WARN  (jetty-launcher-64423-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2748004 INFO  (jetty-launcher-64423-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2748004 INFO  (jetty-launcher-64423-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2748004 INFO  (jetty-launcher-64423-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2748009 INFO  (jetty-launcher-64423-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2748009 INFO  (jetty-launcher-64423-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2748010 INFO  (jetty-launcher-64423-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2748012 INFO  (jetty-launcher-64423-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2748012 INFO  (jetty-launcher-64423-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2748013 INFO  (jetty-launcher-64423-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2748013 INFO  (jetty-launcher-64423-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2748013 INFO  (jetty-launcher-64423-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2748013 INFO  (jetty-launcher-64423-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2748013 INFO  (jetty-launcher-64423-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2748013 INFO  (jetty-launcher-64423-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2748013 INFO  (jetty-launcher-64423-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2748014 INFO  (jetty-launcher-64423-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@230fc6c{/solr,null,AVAILABLE}
   [junit4]   2> 2748014 INFO  (jetty-launcher-64423-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@16ec641a{/solr,null,AVAILABLE}
   [junit4]   2> 2748014 WARN  (jetty-launcher-64423-thread-2) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@7e4d107[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2748015 INFO  (jetty-launcher-64423-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f411c2c{/solr,null,AVAILABLE}
   [junit4]   2> 2748015 INFO  (jetty-launcher-64423-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@44eed52c{SSL,[ssl, http/1.1]}{127.0.0.1:37763}
   [junit4]   2> 2748015 INFO  (jetty-launcher-64423-thread-2) [    ] o.e.j.s.Server Started @2748070ms
   [junit4]   2> 2748015 INFO  (jetty-launcher-64423-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37763}
   [junit4]   2> 2748016 WARN  (jetty-launcher-64423-thread-1) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@54fd4ca8[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2748016 ERROR (jetty-launcher-64423-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2748016 INFO  (jetty-launcher-64423-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2748016 INFO  (jetty-launcher-64423-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 2748016 INFO  (jetty-launcher-64423-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2748016 INFO  (jetty-launcher-64423-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2748016 INFO  (jetty-launcher-64423-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-05T07:05:52.979Z
   [junit4]   2> 2748016 INFO  (jetty-launcher-64423-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@79049260{SSL,[ssl, http/1.1]}{127.0.0.1:42220}
   [junit4]   2> 2748016 INFO  (jetty-launcher-64423-thread-1) [    ] o.e.j.s.Server Started @2748072ms
   [junit4]   2> 2748016 INFO  (jetty-launcher-64423-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42220}
   [junit4]   2> 2748017 ERROR (jetty-launcher-64423-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2748015 WARN  (jetty-launcher-64423-thread-3) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@659661dc[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2748017 INFO  (jetty-launcher-64423-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2748017 INFO  (jetty-launcher-64423-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 2748017 INFO  (jetty-launcher-64423-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2748017 INFO  (jetty-launcher-64423-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2748017 INFO  (jetty-launcher-64423-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-05T07:05:52.980Z
   [junit4]   2> 2748019 INFO  (jetty-launcher-64423-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6a7383e0{SSL,[ssl, http/1.1]}{127.0.0.1:44062}
   [junit4]   2> 2748019 INFO  (jetty-launcher-64423-thread-3) [    ] o.e.j.s.Server Started @2748074ms
   [junit4]   2> 2748019 INFO  (jetty-launcher-64423-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44062}
   [junit4]   2> 2748019 ERROR (jetty-launcher-64423-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2748019 INFO  (jetty-launcher-64423-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2748019 INFO  (jetty-launcher-64423-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 2748019 INFO  (jetty-launcher-64423-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2748019 INFO  (jetty-launcher-64423-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2748019 INFO  (jetty-launcher-64423-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-05T07:05:52.982Z
   [junit4]   2> 2748022 INFO  (zkConnectionManagerCallback-64425-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748022 INFO  (zkConnectionManagerCallback-64429-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748022 INFO  (zkConnectionManagerCallback-64427-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748022 INFO  (jetty-launcher-64423-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2748023 INFO  (jetty-launcher-64423-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2748023 INFO  (jetty-launcher-64423-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2748138 INFO  (jetty-launcher-64423-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43497/solr
   [junit4]   2> 2748155 INFO  (zkConnectionManagerCallback-64433-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748158 INFO  (zkConnectionManagerCallback-64436-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748160 INFO  (jetty-launcher-64423-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43497/solr
   [junit4]   2> 2748161 INFO  (zkConnectionManagerCallback-64439-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748163 INFO  (zkConnectionManagerCallback-64441-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748273 INFO  (jetty-launcher-64423-thread-2) [n:127.0.0.1:37763_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37763_solr
   [junit4]   2> 2748274 INFO  (jetty-launcher-64423-thread-2) [n:127.0.0.1:37763_solr    ] o.a.s.c.Overseer Overseer (id=72569215775277063-127.0.0.1:37763_solr-n_0000000000) starting
   [junit4]   2> 2748283 INFO  (zkConnectionManagerCallback-64450-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748287 INFO  (jetty-launcher-64423-thread-2) [n:127.0.0.1:37763_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43497/solr ready
   [junit4]   2> 2748297 INFO  (jetty-launcher-64423-thread-2) [n:127.0.0.1:37763_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37763_solr
   [junit4]   2> 2748297 INFO  (OverseerStateUpdate-72569215775277063-127.0.0.1:37763_solr-n_0000000000) [n:127.0.0.1:37763_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37763_solr
   [junit4]   2> 2748301 INFO  (zkCallback-64449-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2748302 INFO  (OverseerStateUpdate-72569215775277063-127.0.0.1:37763_solr-n_0000000000) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2748309 INFO  (jetty-launcher-64423-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43497/solr
   [junit4]   2> 2748312 INFO  (zkConnectionManagerCallback-64454-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748316 INFO  (zkConnectionManagerCallback-64456-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748327 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2748331 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2748331 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:44062_solr as DOWN
   [junit4]   2> 2748332 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2748333 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44062_solr
   [junit4]   2> 2748334 INFO  (zkCallback-64435-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2748335 INFO  (zkCallback-64449-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2748335 INFO  (zkCallback-64455-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2748339 INFO  (zkCallback-64440-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2748340 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:42220_solr as DOWN
   [junit4]   2> 2748341 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2748341 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42220_solr
   [junit4]   2> 2748343 INFO  (zkCallback-64449-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2748343 INFO  (zkCallback-64435-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2748344 INFO  (zkCallback-64440-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2748344 INFO  (zkCallback-64455-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2748371 INFO  (jetty-launcher-64423-thread-2) [n:127.0.0.1:37763_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2748372 INFO  (zkConnectionManagerCallback-64464-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748374 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2748374 INFO  (zkConnectionManagerCallback-64469-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748375 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43497/solr ready
   [junit4]   2> 2748376 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2748376 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2748381 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43497/solr ready
   [junit4]   2> 2748382 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2748422 INFO  (jetty-launcher-64423-thread-2) [n:127.0.0.1:37763_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37763.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2748460 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42220.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2748464 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44062.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2748472 INFO  (jetty-launcher-64423-thread-2) [n:127.0.0.1:37763_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37763.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2748473 INFO  (jetty-launcher-64423-thread-2) [n:127.0.0.1:37763_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37763.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2748477 INFO  (jetty-launcher-64423-thread-2) [n:127.0.0.1:37763_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/.
   [junit4]   2> 2748484 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42220.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2748484 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42220.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2748485 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44062.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2748485 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44062.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2748486 INFO  (jetty-launcher-64423-thread-1) [n:127.0.0.1:42220_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/.
   [junit4]   2> 2748487 INFO  (jetty-launcher-64423-thread-3) [n:127.0.0.1:44062_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/.
   [junit4]   2> 2748611 INFO  (zkConnectionManagerCallback-64476-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2748612 INFO  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2748613 INFO  (SUITE-LIROnShardRestartTest-seed#[E36FE4B4C0CCB6C9]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43497/solr ready
   [junit4]   2> 2748654 INFO  (TEST-LIROnShardRestartTest.testAllReplicasInLIR-seed#[E36FE4B4C0CCB6C9]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAllReplicasInLIR
   [junit4]   2> 2748675 INFO  (qtp638623093-181093) [n:127.0.0.1:44062_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&name=allReplicasInLIR&nrtReplicas=3&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2748678 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection allReplicasInLIR
   [junit4]   2> 2748678 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/managed-schema to /configs/allReplicasInLIR.AUTOCREATED/managed-schema
   [junit4]   2> 2748680 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/protwords.txt to /configs/allReplicasInLIR.AUTOCREATED/protwords.txt
   [junit4]   2> 2748682 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/solrconfig.xml to /configs/allReplicasInLIR.AUTOCREATED/solrconfig.xml
   [junit4]   2> 2748683 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/synonyms.txt to /configs/allReplicasInLIR.AUTOCREATED/synonyms.txt
   [junit4]   2> 2748685 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/stopwords.txt to /configs/allReplicasInLIR.AUTOCREATED/stopwords.txt
   [junit4]   2> 2748687 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_it.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/contractions_it.txt
   [junit4]   2> 2748689 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ca.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/contractions_ca.txt
   [junit4]   2> 2748690 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stemdict_nl.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stemdict_nl.txt
   [junit4]   2> 2748692 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_no.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_no.txt
   [junit4]   2> 2748694 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hy.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_hy.txt
   [junit4]   2> 2748695 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_id.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_id.txt
   [junit4]   2> 2748697 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hi.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_hi.txt
   [junit4]   2> 2748699 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ga.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ga.txt
   [junit4]   2> 2748701 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_da.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_da.txt
   [junit4]   2> 2748702 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ja.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ja.txt
   [junit4]   2> 2748704 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_pt.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_pt.txt
   [junit4]   2> 2748706 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hu.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_hu.txt
   [junit4]   2> 2748708 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_el.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_el.txt
   [junit4]   2> 2748710 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ru.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ru.txt
   [junit4]   2> 2748711 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_tr.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_tr.txt
   [junit4]   2> 2748713 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ar.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ar.txt
   [junit4]   2> 2748715 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/userdict_ja.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/userdict_ja.txt
   [junit4]   2> 2748716 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_eu.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_eu.txt
   [junit4]   2> 2748718 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_it.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_it.txt
   [junit4]   2> 2748720 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_cz.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_cz.txt
   [junit4]   2> 2748722 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stoptags_ja.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stoptags_ja.txt
   [junit4]   2> 2748728 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_fr.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/contractions_fr.txt
   [junit4]   2> 2748730 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_de.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_de.txt
   [junit4]   2> 2748732 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fa.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_fa.txt
   [junit4]   2> 2748735 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ga.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/contractions_ga.txt
   [junit4]   2> 2748737 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ca.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ca.txt
   [junit4]   2> 2748739 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_nl.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_nl.txt
   [junit4]   2> 2748740 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_bg.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_bg.txt
   [junit4]   2> 2748742 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_en.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_en.txt
   [junit4]   2> 2748745 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_es.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_es.txt
   [junit4]   2> 2748747 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_lv.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_lv.txt
   [junit4]   2> 2748749 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ro.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_ro.txt
   [junit4]   2> 2748751 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_th.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_th.txt
   [junit4]   2> 2748753 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fr.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_fr.txt
   [junit4]   2> 2748755 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_sv.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_sv.txt
   [junit4]   2> 2748757 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fi.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_fi.txt
   [junit4]   2> 2748758 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/hyphenations_ga.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/hyphenations_ga.txt
   [junit4]   2> 2748760 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_gl.txt to /configs/allReplicasInLIR.AUTOCREATED/lang/stopwords_gl.txt
   [junit4]   2> 2748762 INFO  (OverseerThreadFactory-13924-thread-1-processing-n:127.0.0.1:37763_solr) [n:127.0.0.1:37763_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/params.json to /configs/allReplicasInLIR.AUTOCREATED/params.json
   [junit4]   2> 2748875 INFO  (qtp638623093-181091) [n:127.0.0.1:44062_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748876 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748877 INFO  (qtp638623093-181089) [n:127.0.0.1:44062_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748878 INFO  (qtp638623093-181090) [n:127.0.0.1:44062_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748879 INFO  (qtp638623093-181091) [n:127.0.0.1:44062_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748901 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748903 INFO  (qtp1372714890-181081) [n:127.0.0.1:37763_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748904 INFO  (qtp1372714890-181082) [n:127.0.0.1:37763_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748905 INFO  (qtp1372714890-181083) [n:127.0.0.1:37763_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748906 INFO  (qtp1372714890-181084) [n:127.0.0.1:37763_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748923 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748925 INFO  (qtp1153914731-181075) [n:127.0.0.1:42220_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748926 INFO  (qtp1153914731-181076) [n:127.0.0.1:42220_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748927 INFO  (qtp1153914731-181078) [n:127.0.0.1:42220_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748927 INFO  (qtp1153914731-181094) [n:127.0.0.1:42220_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2748934 INFO  (OverseerStateUpdate-72569215775277063-127.0.0.1:37763_solr-n_0000000000) [n:127.0.0.1:37763_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"allReplicasInLIR",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"allReplicasInLIR_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:42220/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2748940 INFO  (OverseerStateUpdate-72569215775277063-127.0.0.1:37763_solr-n_0000000000) [n:127.0.0.1:37763_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"allReplicasInLIR",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"allReplicasInLIR_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37763/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2748954 INFO  (OverseerStateUpdate-72569215775277063-127.0.0.1:37763_solr-n_0000000000) [n:127.0.0.1:37763_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"allReplicasInLIR",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"allReplicasInLIR_shard1_replica_n5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:44062/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 2749156 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr    x:allReplicasInLIR_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=allReplicasInLIR.AUTOCREATED&newCollection=true&name=allReplicasInLIR_shard1_replica_n1&action=CREATE&numShards=1&collection=allReplicasInLIR&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2749156 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr    x:allReplicasInLIR_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=allReplicasInLIR.AUTOCREATED&newCollection=true&name=allReplicasInLIR_shard1_replica_n2&action=CREATE&numShards=1&collection=allReplicasInLIR&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2749157 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr    x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2749158 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr    x:allReplicasInLIR_shard1_replica_n5] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=allReplicasInLIR.AUTOCREATED&newCollection=true&name=allReplicasInLIR_shard1_replica_n5&action=CREATE&numShards=1&collection=allReplicasInLIR&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2750181 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
   [junit4]   2> 2750181 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
   [junit4]   2> 2750181 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
   [junit4]   2> 2750181 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 2750181 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 2750181 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 2750182 WARN  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../contrib/extraction/lib
   [junit4]   2> 2750182 WARN  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../dist
   [junit4]   2> 2750183 WARN  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../contrib/clustering/lib
   [junit4]   2> 2750183 WARN  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../dist
   [junit4]   2> 2750183 WARN  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../contrib/langid/lib
   [junit4]   2> 2750183 WARN  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../dist
   [junit4]   2> 2750184 WARN  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../contrib/velocity/lib
   [junit4]   2> 2750184 WARN  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5/../../../../dist
   [junit4]   2> 2750185 WARN  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../contrib/extraction/lib
   [junit4]   2> 2750185 WARN  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../contrib/extraction/lib
   [junit4]   2> 2750186 WARN  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../dist
   [junit4]   2> 2750186 WARN  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../dist
   [junit4]   2> 2750187 WARN  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../contrib/clustering/lib
   [junit4]   2> 2750187 WARN  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../contrib/clustering/lib
   [junit4]   2> 2750187 WARN  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../dist
   [junit4]   2> 2750187 WARN  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../dist
   [junit4]   2> 2750187 WARN  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../contrib/langid/lib
   [junit4]   2> 2750187 WARN  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../contrib/langid/lib
   [junit4]   2> 2750187 WARN  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../dist
   [junit4]   2> 2750188 WARN  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../dist
   [junit4]   2> 2750188 WARN  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../contrib/velocity/lib
   [junit4]   2> 2750188 WARN  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2/../../../../dist
   [junit4]   2> 2750188 WARN  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../contrib/velocity/lib
   [junit4]   2> 2750188 WARN  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1/../../../../dist
   [junit4]   2> 2750206 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 2750209 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 2750211 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 2750234 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.s.IndexSchema [allReplicasInLIR_shard1_replica_n5] Schema name=default-config
   [junit4]   2> 2750237 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.s.IndexSchema [allReplicasInLIR_shard1_replica_n1] Schema name=default-config
   [junit4]   2> 2750239 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.s.IndexSchema [allReplicasInLIR_shard1_replica_n2] Schema name=default-config
   [junit4]   2> 2750496 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 2750504 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 2750509 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 2750550 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'allReplicasInLIR_shard1_replica_n2' using configuration from collection allReplicasInLIR, trusted=true
   [junit4]   2> 2750551 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37763.solr.core.allReplicasInLIR.shard1.replica_n2' (registry 'solr.core.allReplicasInLIR.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2750551 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.c.SolrCore [[allReplicasInLIR_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/allReplicasInLIR_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node3/./allReplicasInLIR_shard1_replica_n2/data/]
   [junit4]   2> 2750574 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 'allReplicasInLIR_shard1_replica_n5' using configuration from collection allReplicasInLIR, trusted=true
   [junit4]   2> 2750574 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44062.solr.core.allReplicasInLIR.shard1.replica_n5' (registry 'solr.core.allReplicasInLIR.shard1.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2750574 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.SolrCore [[allReplicasInLIR_shard1_replica_n5] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/allReplicasInLIR_shard1_replica_n5], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node2/./allReplicasInLIR_shard1_replica_n5/data/]
   [junit4]   2> 2750582 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'allReplicasInLIR_shard1_replica_n1' using configuration from collection allReplicasInLIR, trusted=true
   [junit4]   2> 2750583 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42220.solr.core.allReplicasInLIR.shard1.replica_n1' (registry 'solr.core.allReplicasInLIR.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2b722c42
   [junit4]   2> 2750583 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.SolrCore [[allReplicasInLIR_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/allReplicasInLIR_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.LIROnShardRestartTest_E36FE4B4C0CCB6C9-001/tempDir-001/node1/./allReplicasInLIR_shard1_replica_n1/data/]
   [junit4]   2> 2750615 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
   [junit4]   2> 2750621 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
   [junit4]   2> 2750622 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
   [junit4]   2> 2750697 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2750698 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2750698 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2750698 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2750698 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2750698 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2750701 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 2750701 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 2750701 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2750701 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2750701 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 2750701 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2750705 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@23442368[allReplicasInLIR_shard1_replica_n2] main]
   [junit4]   2> 2750705 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.s.SolrIndexSearcher Opening [Searcher@70ca9336[allReplicasInLIR_shard1_replica_n5] main]
   [junit4]   2> 2750705 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1c46f204[allReplicasInLIR_shard1_replica_n1] main]
   [junit4]   2> 2750706 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/allReplicasInLIR.AUTOCREATED
   [junit4]   2> 2750706 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/allReplicasInLIR.AUTOCREATED
   [junit4]   2> 2750707 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/allReplicasInLIR.AUTOCREATED
   [junit4]   2> 2750707 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/allReplicasInLIR.AUTOCREATED
   [junit4]   2> 2750707 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/allReplicasInLIR.AUTOCREATED/managed-schema
   [junit4]   2> 2750707 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/allReplicasInLIR.AUTOCREATED
   [junit4]   2> 2750707 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/allReplicasInLIR.AUTOCREATED/managed-schema
   [junit4]   2> 2750707 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/allReplicasInLIR.AUTOCREATED
   [junit4]   2> 2750707 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/allReplicasInLIR.AUTOCREATED/managed-schema
   [junit4]   2> 2750707 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 2750708 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 2750708 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 2750709 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2750709 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 2750709 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2750709 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 2750709 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2750709 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 2750712 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2750712 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2750712 INFO  (qtp1153914731-181095) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1621803092593868800
   [junit4]   2> 2750712 INFO  (qtp638623093-181092) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1621803092593868800
   [junit4]   2> 2750712 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2750712 INFO  (qtp1372714890-181086) [n:127.0.0.1:37763_solr c:allReplicasInLIR s:shard1 r:core_node4 x:allReplicasInLIR_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1621803092593868800
   [junit4]   2> 2750715 INFO  (searcherExecutor-13942-thread-1-processing-n:127.0.0.1:42220_solr x:allReplicasInLIR_shard1_replica_n1 c:allReplicasInLIR s:shard1 r:core_node3) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@1c46f204[allReplicasInLIR_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2750715 INFO  (searcherExecutor-13942-thread-1-processing-n:127.0.0.1:42220_solr x:allReplicasInLIR_shard1_replica_n1 c:allReplicasInLIR s:shard1 r:core_node3) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 2750715 INFO  (searcherExecutor-13942-thread-1-processing-n:127.0.0.1:42220_solr x:allReplicasInLIR_shard1_replica_n1 c:allReplicasInLIR s:shard1 r:core_node3) [n:127.0.0.1:42220_solr c:allReplicasInLIR s:shard1 r:core_node3 x:allReplicasInLIR_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 2750715 INFO  (searcherExecutor-13941-thread-1-processing-n:127.0.0.1:44062_solr x:allReplicasInLIR_shard1_replica_n5 c:allReplicasInLIR s:shard1 r:core_node6) [n:127.0.0.1:44062_solr c:allReplicasInLIR s:shard1 r:core_node6 x:allReplicasInLIR_shard1_replica_n5] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@70ca9336[allReplicasInLIR_shard1_replica_n5] main{ExitableDirectoryR

[...truncated too long message...]

vy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1189672334
     [copy] Copying 242 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1189672334
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1189672334

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 41ms :: artifacts dl 4ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 328 minutes 34 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 424 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/424/

6 tests failed.
FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:41894/mo_/vt/forceleader_test_collection

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:41894/mo_/vt/forceleader_test_collection
        at __randomizedtesting.SeedInfo.seed([D2A541849C903D90:34327544A512C4F1]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:294)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 52 more


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at __randomizedtesting.SeedInfo.seed([D2A541849C903D90:F95A94DF06E82840]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeLostTriggerRestoreState(TestSimTriggerIntegration.java:332)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:508)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:959)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)  at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:770)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)  at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)   expected null, but was:<ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:508)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:959)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)  at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:770)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)  at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  >

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:508)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:959)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:770)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

 expected null, but was:<ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:508)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:959)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.NRTCachingDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:770)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

>
        at __randomizedtesting.SeedInfo.seed([D2A541849C903D90]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotNull(Assert.java:755)
        at org.junit.Assert.assertNull(Assert.java:737)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:317)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
        at __randomizedtesting.SeedInfo.seed([D2A541849C903D90:5AF17E5E326C5068]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:195)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:143)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:138)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1008)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:164)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:108)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.LIROnShardRestartTest.testAllReplicasInLIR

Error Message:
Path must not end with / character

Stack Trace:
java.lang.IllegalArgumentException: Path must not end with / character
        at __randomizedtesting.SeedInfo.seed([D2A541849C903D90:883D7B42E2105A77]:0)
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:58)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1523)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346)
        at org.apache.solr.cloud.LIROnShardRestartTest.testAllReplicasInLIR(LIROnShardRestartTest.java:168)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.LIROnShardRestartTest.testSeveralReplicasInLIR

Error Message:
Error from server at http://127.0.0.1:39372/solr: Cannot create collection severalReplicasInLIR. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 2. This allows a maximum of 2 to be created. Value of numShards is 1, value of nrtReplicas is 3, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 3 shards to be created (higher than the allowed number)

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:39372/solr: Cannot create collection severalReplicasInLIR. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 2. This allows a maximum of 2 to be created. Value of numShards is 1, value of nrtReplicas is 3, value of tlogReplicas is 0 and value of pullReplicas is 0. This requires 3 shards to be created (higher than the allowed number)
        at __randomizedtesting.SeedInfo.seed([D2A541849C903D90:54FFD6B0BCCA70C9]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.LIROnShardRestartTest.testSeveralReplicasInLIR(LIROnShardRestartTest.java:190)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13285 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
   [junit4]   2> 157108 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_D2A541849C903D90-001/init-core-data-001
   [junit4]   2> 157109 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 157109 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 157111 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 157113 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 157889 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 159089 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 159337 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.m.log Logging to org.apache.logging.slf4j.Log4jLogger@43d1b38c via org.mortbay.log.Slf4jLog
   [junit4]   2> 159382 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 159958 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 160036 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_34408_hdfs____.fxtq1n/webapp
   [junit4]   2> 161069 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34408
   [junit4]   2> 162419 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 162426 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 162451 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_42283_datanode____.zbsw16/webapp
   [junit4]   2> 163078 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42283
   [junit4]   2> 163748 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 163751 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 163800 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_34982_datanode____xr10bd/webapp
   [junit4]   2> 164534 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[D2A541849C903D90]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34982
   [junit4]   2> 165060 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_D2A541849C903D90-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_D2A541849C903D90-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:40890) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 165062 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_D2A541849C903D90-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_D2A541849C903D90-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:40890) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 165302 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x1bffdb37caaaa8: from storage DS-2f50ffa3-addd-43e0-b0b3-811c380fbe4e node DatanodeRegistration(127.0.0.1:40281, datanodeUuid=8a2ad3e7-20d7-462b-8102-91962ed25de9, infoPort=38802, infoSecurePort=0, ipcPort=34266, storageInfo=lv=-56;cid=testClusterID;nsid=848385282;c=0), blocks: 0, hasStaleStorage: true, processing time: 4 msecs
   [junit4]   2> 165302 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x1bffdb37ca62fb: from storage DS-dc18ac7e-9c91-481b-a185-b2d6fae6e7bf node DatanodeRegistration(127.0.0.1:36966, datanodeUuid=e7f7a922-d56a-472f-aa9a-5be876c3d3ec, infoPort=42695, infoSecurePort=0, ipcPort=42964, storageInfo=lv=-56;cid=testClusterID;nsid=848385282;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 165312 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x1bffdb37caaaa8: from storage DS-f838ede3-dad4-468e-ae49-bf22bc84b5bb node DatanodeRegistration(127.0.0.1:40281, datanodeUuid=8a2ad3e7-20d7-462b-8102-91962ed25de9, infoPort=38802, infoSecurePort=0, ipcPort=34266, storageInfo=lv=-56;cid=testClusterID;nsid=848385282;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 165312 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x1bffdb37ca62fb: from storage DS-d9b5bcbc-dc12-48e2-9317-a7b9894a830e node DatanodeRegistration(127.0.0.1:36966, datanodeUuid=e7f7a922-d56a-472f-aa9a-5be876c3d3ec, infoPort=42695, infoSecurePort=0, ipcPort=42964, storageInfo=lv=-56;cid=testClusterID;nsid=848385282;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 165850 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 165853 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 165853 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 165953 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer start zk server on port:42738
   [junit4]   2> 165953 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:42738
   [junit4]   2> 165953 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 42738
   [junit4]   2> 165964 INFO  (zkConnectionManagerCallback-317-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 165970 INFO  (zkConnectionManagerCallback-319-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 165976 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 165978 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 165980 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 165981 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 165983 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 165986 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 165988 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 165989 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 165991 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 165993 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 165994 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 166011 INFO  (zkConnectionManagerCallback-323-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 166022 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x101d5c0aebe0002, likely client has closed socket
   [junit4]   2> 166023 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 166235 WARN  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 166236 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 166236 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 166236 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 166237 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 166237 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 166237 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 166238 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@54417188{/,null,AVAILABLE}
   [junit4]   2> 166239 WARN  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@3591a942[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 166253 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2f126c72{SSL,[ssl, http/1.1]}{127.0.0.1:37222}
   [junit4]   2> 166253 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.e.j.s.Server Started @166322ms
   [junit4]   2> 166253 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:40890/hdfs__localhost_40890__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-7.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_D2A541849C903D90-001_tempDir-002_control_data, replicaType=NRT, hostContext=/, hostPort=37222, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_D2A541849C903D90-001/control-001/cores}
   [junit4]   2> 166254 ERROR (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 166254 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 166254 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 166254 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 166254 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 166254 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-06T03:46:07.636Z
   [junit4]   2> 166267 INFO  (zkConnectionManagerCallback-325-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 166270 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 166270 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_D2A541849C903D90-001/control-001/solr.xml
   [junit4]   2> 166281 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 166281 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 166284 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@58171695, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 166439 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42738/solr
   [junit4]   2> 166449 INFO  (zkConnectionManagerCallback-329-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 166474 INFO  (zkConnectionManagerCallback-331-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 166655 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [n:127.0.0.1:37222_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37222_
   [junit4]   2> 166656 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [n:127.0.0.1:37222_    ] o.a.s.c.Overseer Overseer (id=72574092556763141-127.0.0.1:37222_-n_0000000000) starting
   [junit4]   2> 166667 INFO  (zkConnectionManagerCallback-338-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 166674 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [n:127.0.0.1:37222_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42738/solr ready
   [junit4]   2> 166676 INFO  (OverseerStateUpdate-72574092556763141-127.0.0.1:37222_-n_0000000000) [n:127.0.0.1:37222_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37222_
   [junit4]   2> 166677 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [n:127.0.0.1:37222_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37222_
   [junit4]   2> 166679 INFO  (zkCallback-337-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 166684 INFO  (zkCallback-330-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 166714 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [n:127.0.0.1:37222_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 166749 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [n:127.0.0.1:37222_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58171695
   [junit4]   2> 166771 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [n:127.0.0.1:37222_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58171695
   [junit4]   2> 166771 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [n:127.0.0.1:37222_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58171695
   [junit4]   2> 166774 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [n:127.0.0.1:37222_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_D2A541849C903D90-001/control-001/cores
   [junit4]   2> 166835 INFO  (zkConnectionManagerCallback-344-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 166837 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 166838 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[D2A541849C903D90]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42738/solr ready
   [junit4]   2> 166872 INFO  (qtp2145808783-1373) [n:127.0.0.1:37222_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37222_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 166877 INFO  (OverseerThreadFactory-633-thread-1-processing-n:127.0.0.1:37222_) [n:127.0.0.1:37222_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 167000 INFO  (qtp2145808783-1371) [n:127.0.0.1:37222_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 167003 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 167004 INFO  (qtp2145808783-1369) [n:127.0.0.1:37222_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 167006 INFO  (qtp2145808783-1370) [n:127.0.0.1:37222_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 167010 INFO  (qtp2145808783-1371) [n:127.0.0.1:37222_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 167027 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 167028 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 168054 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 168083 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 168220 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 168252 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 168252 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58171695
   [junit4]   2> 168267 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40890/solr_hdfs_home
   [junit4]   2> 168267 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 168268 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_D2A541849C903D90-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://localhost:40890/solr_hdfs_home/control_collection/core_node2/data/]
   [junit4]   2> 168270 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40890/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 168312 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 168312 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 168312 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 169278 WARN  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 174690 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 174700 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40890/solr_hdfs_home/control_collection/core_node2/data
   [junit4]   2> 174782 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40890/solr_hdfs_home/control_collection/core_node2/data/index
   [junit4]   2> 174801 INFO  (qtp2145808783-1372) [n:127.0.0.1:37222_ c:control_collection s:shard1  x:control_collection_shard1_

[...truncated too long message...]



ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null254807252
     [copy] Copying 242 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null254807252
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null254807252

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 55ms :: artifacts dl 4ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 333 minutes 16 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 425 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/425/

2 tests failed.
FAILED:  org.apache.lucene.document.TestLatLonLineShapeQueries.testRandomBig

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([B61B9140D7CA01CB:314CECCF46937D4B]:0)
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
        at java.lang.StringBuilder.append(StringBuilder.java:136)
        at org.apache.lucene.index.CorruptIndexException.<init>(CorruptIndexException.java:62)
        at org.apache.lucene.index.CorruptIndexException.<init>(CorruptIndexException.java:47)
        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:471)
        at org.apache.lucene.util.bkd.BKDWriter.verifyChecksum(BKDWriter.java:1429)
        at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1884)
        at org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:1024)
        at org.apache.lucene.index.RandomCodec$1$1.writeField(RandomCodec.java:142)
        at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:62)
        at org.apache.lucene.codecs.PointsWriter.merge(PointsWriter.java:191)
        at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:145)
        at org.apache.lucene.codecs.asserting.AssertingPointsFormat$AssertingPointsWriter.merge(AssertingPointsFormat.java:142)
        at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:188)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2177)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5139)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1619)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
        at org.apache.lucene.document.BaseLatLonShapeTestCase.indexRandomShapes(BaseLatLonShapeTestCase.java:248)
        at org.apache.lucene.document.BaseLatLonShapeTestCase.verify(BaseLatLonShapeTestCase.java:224)
        at org.apache.lucene.document.BaseLatLonShapeTestCase.doTestRandom(BaseLatLonShapeTestCase.java:205)
        at org.apache.lucene.document.BaseLatLonShapeTestCase.testRandomBig(BaseLatLonShapeTestCase.java:181)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)


FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:37517/xh_na/ct/forceleader_test_collection

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:37517/xh_na/ct/forceleader_test_collection
        at __randomizedtesting.SeedInfo.seed([82FC50136F61DD87:646B64D356E324E6]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:294)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 52 more




Build Log:
[...truncated 10360 lines...]
   [junit4] Suite: org.apache.lucene.document.TestLatLonLineShapeQueries
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLatLonLineShapeQueries -Dtests.method=testRandomBig -Dtests.seed=B61B9140D7CA01CB -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ar-MA -Dtests.timezone=Asia/Damascus -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    154s J2 | TestLatLonLineShapeQueries.testRandomBig <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([B61B9140D7CA01CB:314CECCF46937D4B]:0)
   [junit4]    > at java.util.Arrays.copyOf(Arrays.java:3332)
   [junit4]    > at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
   [junit4]    > at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
   [junit4]    > at java.lang.StringBuilder.append(StringBuilder.java:136)
   [junit4]    > at org.apache.lucene.index.CorruptIndexException.<init>(CorruptIndexException.java:62)
   [junit4]    > at org.apache.lucene.index.CorruptIndexException.<init>(CorruptIndexException.java:47)
   [junit4]    > at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:471)
   [junit4]    > at org.apache.lucene.util.bkd.BKDWriter.verifyChecksum(BKDWriter.java:1429)
   [junit4]    > at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1884)
   [junit4]    > at org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:1024)
   [junit4]    > at org.apache.lucene.index.RandomCodec$1$1.writeField(RandomCodec.java:142)
   [junit4]    > at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:62)
   [junit4]    > at org.apache.lucene.codecs.PointsWriter.merge(PointsWriter.java:191)
   [junit4]    > at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:145)
   [junit4]    > at org.apache.lucene.codecs.asserting.AssertingPointsFormat$AssertingPointsWriter.merge(AssertingPointsFormat.java:142)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:188)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
   [junit4]    > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
   [junit4]    > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
   [junit4]    > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]    > at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2177)
   [junit4]    > at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5139)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1619)
   [junit4]    > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
   [junit4]    > at org.apache.lucene.document.BaseLatLonShapeTestCase.indexRandomShapes(BaseLatLonShapeTestCase.java:248)
   [junit4]    > at org.apache.lucene.document.BaseLatLonShapeTestCase.verify(BaseLatLonShapeTestCase.java:224)
   [junit4]    > at org.apache.lucene.document.BaseLatLonShapeTestCase.doTestRandom(BaseLatLonShapeTestCase.java:205)
   [junit4]    > at org.apache.lucene.document.BaseLatLonShapeTestCase.testRandomBig(BaseLatLonShapeTestCase.java:181)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/sandbox/test/J2/temp/lucene.document.TestLatLonLineShapeQueries_B61B9140D7CA01CB-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))}, docValues:{shape=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1344, maxMBSortInHeap=6.6523539447183815, sim=RandomSimilarity(queryNorm=false): {}, locale=ar-MA, timezone=Asia/Damascus
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=359530632,total=371720192
   [junit4]   2> NOTE: All tests run in this JVM: [TestLatLonShape, TestLatLonLineShapeQueries]
   [junit4] Completed [17/23 (1!)] on J2 in 688.17s, 4 tests, 1 error <<< FAILURES!

[...truncated 11 lines...]
   [junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/sandbox/test/temp/junit4-J2-20190107_032426_2158196766380210708550.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid24753.hprof ...
   [junit4] Heap dump file created [430894630 bytes in 0.764 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 4127 lines...]
   [junit4] Suite: org.apache.solr.cloud.ForceLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/init-core-data-001
   [junit4]   2> 2378084 WARN  (SUITE-ForceLeaderTest-seed#[82FC50136F61DD87]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 2378085 INFO  (SUITE-ForceLeaderTest-seed#[82FC50136F61DD87]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2378087 INFO  (SUITE-ForceLeaderTest-seed#[82FC50136F61DD87]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 2378087 INFO  (SUITE-ForceLeaderTest-seed#[82FC50136F61DD87]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2378087 INFO  (SUITE-ForceLeaderTest-seed#[82FC50136F61DD87]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /xh_na/ct
   [junit4]   2> 2378091 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2378091 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2378091 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2378191 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer start zk server on port:44374
   [junit4]   2> 2378191 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:44374
   [junit4]   2> 2378191 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 44374
   [junit4]   2> 2378206 INFO  (zkConnectionManagerCallback-10847-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2378219 INFO  (zkConnectionManagerCallback-10849-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2378221 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2378223 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2378228 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2378229 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2378230 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2378231 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2378233 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2378234 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2378235 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2378236 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2378238 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2378239 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2378417 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2378417 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2378417 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2378417 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2378423 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2378423 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2378423 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2378426 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@436f6a44{/xh_na/ct,null,AVAILABLE}
   [junit4]   2> 2378427 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@155fd3ea{HTTP/1.1,[http/1.1]}{127.0.0.1:44391}
   [junit4]   2> 2378427 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.e.j.s.Server Started @2378538ms
   [junit4]   2> 2378427 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/tempDir-001/control/data, replicaType=NRT, hostContext=/xh_na/ct, hostPort=42207, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/control-001/cores}
   [junit4]   2> 2378428 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2378428 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2378428 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2378428 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2378428 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2378428 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-07T05:09:28.004Z
   [junit4]   2> 2378435 INFO  (zkConnectionManagerCallback-10851-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2378437 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2378437 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/control-001/solr.xml
   [junit4]   2> 2378522 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2378522 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2378524 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2379147 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44374/solr
   [junit4]   2> 2379150 INFO  (zkConnectionManagerCallback-10855-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2379153 INFO  (zkConnectionManagerCallback-10857-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2379247 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42207_xh_na%2Fct
   [junit4]   2> 2379247 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.Overseer Overseer (id=72580082575409156-127.0.0.1:42207_xh_na%2Fct-n_0000000000) starting
   [junit4]   2> 2379254 INFO  (zkConnectionManagerCallback-10864-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2379256 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44374/solr ready
   [junit4]   2> 2379257 INFO  (OverseerStateUpdate-72580082575409156-127.0.0.1:42207_xh_na%2Fct-n_0000000000) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:42207_xh_na%2Fct
   [junit4]   2> 2379257 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42207_xh_na%2Fct
   [junit4]   2> 2379259 INFO  (zkCallback-10856-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2379259 INFO  (zkCallback-10863-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2379289 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2379319 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2379334 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2379335 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2379337 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/control-001/cores
   [junit4]   2> 2379367 INFO  (zkConnectionManagerCallback-10870-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2379369 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2379370 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44374/solr ready
   [junit4]   2> 2379370 INFO  (SocketProxy-Acceptor-42207) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=58916,localport=42207], receiveBufferSize:531000
   [junit4]   2> 2379371 INFO  (SocketProxy-Acceptor-42207) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=44391,localport=56778], receiveBufferSize=530904
   [junit4]   2> 2379372 INFO  (qtp72007760-48979) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42207_xh_na%252Fct&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2379380 INFO  (OverseerThreadFactory-11696-thread-1-processing-n:127.0.0.1:42207_xh_na%2Fct) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2379486 INFO  (SocketProxy-Acceptor-42207) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=58920,localport=42207], receiveBufferSize:531000
   [junit4]   2> 2379486 INFO  (SocketProxy-Acceptor-42207) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=44391,localport=56782], receiveBufferSize=530904
   [junit4]   2> 2379487 INFO  (qtp72007760-48981) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2379488 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2379489 INFO  (qtp72007760-48978) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2379489 INFO  (qtp72007760-48980) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2379490 INFO  (qtp72007760-48981) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2379494 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2379494 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2380514 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 2380530 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2380726 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2380751 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2380752 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2380752 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2380756 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1568650642, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2380759 WARN  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2380826 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2380826 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2380829 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2380829 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2380831 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=74.5546875, floorSegmentMB=0.8095703125, forceMergeDeletesPctAllowed=18.076127756227713, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=40.319717673350375
   [junit4]   2> 2380832 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3e29cf85[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2380833 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2380833 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2380834 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2380834 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1621976961945436160
   [junit4]   2> 2380838 INFO  (searcherExecutor-11701-thread-1-processing-n:127.0.0.1:42207_xh_na%2Fct x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@3e29cf85[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2380841 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 2380841 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 2380846 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2380846 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2380846 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:42207/xh_na/ct/control_collection_shard1_replica_n1/
   [junit4]   2> 2380846 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2380847 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:42207/xh_na/ct/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2380847 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2380847 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/72580082575409156-core_node2-n_0000000000
   [junit4]   2> 2380850 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:42207/xh_na/ct/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2380951 INFO  (zkCallback-10856-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2380952 INFO  (zkCallback-10856-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2380952 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2380955 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1461
   [junit4]   2> 2380958 INFO  (qtp72007760-48979) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2381060 INFO  (zkCallback-10856-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2381060 INFO  (zkCallback-10856-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2381060 INFO  (zkCallback-10856-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2381061 INFO  (qtp72007760-48979) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42207_xh_na%252Fct&wt=javabin&version=2} status=0 QTime=1689
   [junit4]   2> 2381081 INFO  (zkConnectionManagerCallback-10875-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2381083 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2381084 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44374/solr ready
   [junit4]   2> 2381084 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2381085 INFO  (SocketProxy-Acceptor-42207) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=58930,localport=42207], receiveBufferSize:531000
   [junit4]   2> 2381086 INFO  (SocketProxy-Acceptor-42207) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=44391,localport=56792], receiveBufferSize=530904
   [junit4]   2> 2381086 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2381090 INFO  (OverseerThreadFactory-11696-thread-2-processing-n:127.0.0.1:42207_xh_na%2Fct) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2381090 INFO  (OverseerCollectionConfigSetProcessor-72580082575409156-127.0.0.1:42207_xh_na%2Fct-n_0000000000) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2381300 WARN  (OverseerThreadFactory-11696-thread-2-processing-n:127.0.0.1:42207_xh_na%2Fct) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2381303 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2381303 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=217
   [junit4]   2> 2381304 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=3
   [junit4]   2> 2381478 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-1-001 of type NRT
   [junit4]   2> 2381478 WARN  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2381479 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2381479 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2381479 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2381480 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2381480 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2381480 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2381480 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6331ba0a{/xh_na/ct,null,AVAILABLE}
   [junit4]   2> 2381481 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2127f5de{HTTP/1.1,[http/1.1]}{127.0.0.1:40424}
   [junit4]   2> 2381481 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.Server Started @2381591ms
   [junit4]   2> 2381481 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/xh_na/ct, hostPort=36331, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-1-001/cores}
   [junit4]   2> 2381481 ERROR (closeThreadPool-10876-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2381481 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2381481 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2381481 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2381481 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2381481 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-07T05:09:31.057Z
   [junit4]   2> 2381483 INFO  (zkConnectionManagerCallback-10878-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2381484 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2381484 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-1-001/solr.xml
   [junit4]   2> 2381490 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2381490 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2381492 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2381564 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44374/solr
   [junit4]   2> 2381565 INFO  (zkConnectionManagerCallback-10882-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2381568 INFO  (zkConnectionManagerCallback-10884-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2381573 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2381578 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.c.ZkController Publish node=127.0.0.1:36331_xh_na%2Fct as DOWN
   [junit4]   2> 2381579 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2381579 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36331_xh_na%2Fct
   [junit4]   2> 2381580 INFO  (zkCallback-10863-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2381580 INFO  (zkCallback-10856-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2381581 INFO  (zkCallback-10883-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2381586 INFO  (zkCallback-10874-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2381613 INFO  (zkConnectionManagerCallback-10891-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2381617 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2381619 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44374/solr ready
   [junit4]   2> 2381619 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2381687 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2381705 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2381705 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2381707 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-2-001 of type NRT
   [junit4]   2> 2381707 INFO  (closeThreadPool-10876-thread-1) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-1-001/cores
   [junit4]   2> 2381708 WARN  (closeThreadPool-10876-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2381708 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2381708 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2381708 INFO  (closeThreadPool-10876-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2381709 INFO  (closeThreadPool-10876-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2381709 INFO  (closeThreadPool-10876-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2381709 INFO  (closeThreadPool-10876-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2381710 INFO  (closeThreadPool-10876-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@55b76a6b{/xh_na/ct,null,AVAILABLE}
   [junit4]   2> 2381710 INFO  (closeThreadPool-10876-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@68709752{HTTP/1.1,[http/1.1]}{127.0.0.1:40183}
   [junit4]   2> 2381710 INFO  (closeThreadPool-10876-thread-2) [    ] o.e.j.s.Server Started @2381821ms
   [junit4]   2> 2381711 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/xh_na/ct, hostPort=39603, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-2-001/cores}
   [junit4]   2> 2381711 ERROR (closeThreadPool-10876-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2381711 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2381711 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2381711 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2381711 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2381711 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-07T05:09:31.287Z
   [junit4]   2> 2381713 INFO  (zkConnectionManagerCallback-10894-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2381714 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2381714 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-2-001/solr.xml
   [junit4]   2> 2381719 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2381719 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2381721 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2381820 INFO  (closeThreadPool-10876-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44374/solr
   [junit4]   2> 2381843 INFO  (zkConnectionManagerCallback-10898-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2381899 INFO  (zkConnectionManagerCallback-10900-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2381904 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2381908 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.c.ZkController Publish node=127.0.0.1:39603_xh_na%2Fct as DOWN
   [junit4]   2> 2381909 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2381909 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39603_xh_na%2Fct
   [junit4]   2> 2381911 INFO  (zkCallback-10874-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2381911 INFO  (zkCallback-10863-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2381911 INFO  (zkCallback-10883-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2381911 INFO  (zkCallback-10856-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2381919 INFO  (zkCallback-10890-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2381928 INFO  (zkCallback-10899-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2381998 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[82FC50136F61DD87]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-3-001 of type NRT
   [junit4]   2> 2381998 WARN  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2381999 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2381999 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2381999 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2382007 INFO  (zkConnectionManagerCallback-10907-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2382009 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2382010 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44374/solr ready
   [junit4]   2> 2382071 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2382098 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2382098 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2382098 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2382103 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@25677177{/xh_na/ct,null,AVAILABLE}
   [junit4]   2> 2382104 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3b3e554d{HTTP/1.1,[http/1.1]}{127.0.0.1:33707}
   [junit4]   2> 2382104 INFO  (closeThreadPool-10876-thread-1) [    ] o.e.j.s.Server Started @2382214ms
   [junit4]   2> 2382104 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/xh_na/ct, hostPort=37517, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-3-001/cores}
   [junit4]   2> 2382104 ERROR (closeThreadPool-10876-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2382104 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2382104 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 2382104 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2382104 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2382105 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-07T05:09:31.681Z
   [junit4]   2> 2382106 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2382107 INFO  (zkConnectionManagerCallback-10910-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2382108 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2382108 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-3-001/solr.xml
   [junit4]   2> 2382112 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2382112 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2382114 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2382122 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2382122 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2382124 INFO  (closeThreadPool-10876-thread-2) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-2-001/cores
   [junit4]   2> 2382255 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44374/solr
   [junit4]   2> 2382256 INFO  (zkConnectionManagerCallback-10914-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2382260 INFO  (zkConnectionManagerCallback-10916-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2382266 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2382271 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.ZkController Publish node=127.0.0.1:37517_xh_na%2Fct as DOWN
   [junit4]   2> 2382272 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2382272 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37517_xh_na%2Fct
   [junit4]   2> 2382274 INFO  (zkCallback-10874-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2382274 INFO  (zkCallback-10890-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2382274 INFO  (zkCallback-10899-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2382274 INFO  (zkCallback-10883-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2382274 INFO  (zkCallback-10856-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2382275 INFO  (zkCallback-10863-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2382275 INFO  (zkCallback-10915-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2382275 INFO  (zkCallback-10906-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2382313 INFO  (zkConnectionManagerCallback-10923-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2382315 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2382316 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44374/solr ready
   [junit4]   2> 2382321 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2382350 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2382368 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2382368 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2382371 INFO  (closeThreadPool-10876-thread-1) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-3-001/cores
   [junit4]   2> 2382460 INFO  (SocketProxy-Acceptor-36331) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=33148,localport=36331], receiveBufferSize:531000
   [junit4]   2> 2382471 INFO  (SocketProxy-Acceptor-36331) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40424,localport=47408], receiveBufferSize=530904
   [junit4]   2> 2382472 INFO  (SocketProxy-Acceptor-36331) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=33152,localport=36331], receiveBufferSize:531000
   [junit4]   2> 2382472 INFO  (qtp2095660133-49045) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:36331_xh_na%252Fct&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2382473 INFO  (SocketProxy-Acceptor-36331) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40424,localport=47414], receiveBufferSize=530904
   [junit4]   2> 2382473 INFO  (SocketProxy-Acceptor-36331) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=33154,localport=36331], receiveBufferSize:531000
   [junit4]   2> 2382474 INFO  (qtp2095660133-49048) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:39603_xh_na%252Fct&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2382474 INFO  (SocketProxy-Acceptor-36331) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40424,localport=47416], receiveBufferSize=530904
   [junit4]   2> 2382474 INFO  (qtp2095660133-49046) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:37517_xh_na%252Fct&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2382481 INFO  (OverseerCollectionConfigSetProcessor-72580082575409156-127.0.0.1:42207_xh_na%2Fct-n_0000000000) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2382494 INFO  (SocketProxy-Acceptor-36331) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=33160,localport=36331], receiveBufferSize:531000
   [junit4]   2> 2382496 INFO  (SocketProxy-Acceptor-36331) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40424,localport=47420], receiveBufferSize=530904
   [junit4]   2> 2382497 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382498 INFO  (qtp2095660133-49047) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382501 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382502 INFO  (qtp2095660133-49047) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382503 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382503 INFO  (SocketProxy-Acceptor-37517) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=49548,localport=37517], receiveBufferSize:531000
   [junit4]   2> 2382505 INFO  (SocketProxy-Acceptor-37517) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=33707,localport=58872], receiveBufferSize=530904
   [junit4]   2> 2382506 INFO  (qtp1430358138-49110) [n:127.0.0.1:37517_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382510 INFO  (qtp1430358138-49111) [n:127.0.0.1:37517_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382511 INFO  (qtp1430358138-49112) [n:127.0.0.1:37517_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382512 INFO  (qtp1430358138-49113) [n:127.0.0.1:37517_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382513 INFO  (qtp1430358138-49108) [n:127.0.0.1:37517_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382514 INFO  (SocketProxy-Acceptor-39603) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57610,localport=39603], receiveBufferSize:531000
   [junit4]   2> 2382514 INFO  (SocketProxy-Acceptor-39603) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40183,localport=51372], receiveBufferSize=530904
   [junit4]   2> 2382515 INFO  (qtp311316050-49078) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382516 INFO  (qtp311316050-49077) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382517 INFO  (qtp311316050-49079) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382518 INFO  (qtp311316050-49080) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382519 INFO  (qtp311316050-49076) [n:127.0.0.1:39603_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382520 INFO  (qtp72007760-48979) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 2382520 INFO  (qtp72007760-48978) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 2382521 INFO  (qtp72007760-48980) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 2382522 INFO  (qtp72007760-48981) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382523 INFO  (qtp72007760-48982) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382524 INFO  (qtp72007760-48979) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382525 INFO  (qtp72007760-48978) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382526 INFO  (qtp72007760-48980) [n:127.0.0.1:42207_xh_na%2Fct    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2382528 INFO  (OverseerThreadFactory-11696-thread-3-processing-n:127.0.0.1:42207_xh_na%2Fct) [n:127.0.0.1:42207_xh_na%2Fct c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:36331_xh_na%2Fct for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 2382529 INFO  (OverseerThreadFactory-11696-thread-4-processing-n:127.0.0.1:42207_xh_na%2Fct) [n:127.0.0.1:42207_xh_na%2Fct c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:39603_xh_na%2Fct for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 2382530 INFO  (OverseerThreadFactory-11696-thread-3-processing-n:127.0.0.1:42207_xh_na%2Fct) [n:127.0.0.1:42207_xh_na%2Fct c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 2382531 INFO  (SocketProxy-Acceptor-36331) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=33172,localport=36331], receiveBufferSize:531000
   [junit4]   2> 2382532 INFO  (OverseerThreadFactory-11696-thread-4-processing-n:127.0.0.1:42207_xh_na%2Fct) [n:127.0.0.1:42207_xh_na%2Fct c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 2382540 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct    x:collection1_shard2_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2382541 INFO  (SocketProxy-Acceptor-39603) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57618,localport=39603], receiveBufferSize:531000
   [junit4]   2> 2382541 INFO  (SocketProxy-Acceptor-36331) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40424,localport=47432], receiveBufferSize=530904
   [junit4]   2> 2382542 INFO  (SocketProxy-Acceptor-39603) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=40183,localport=51380], receiveBufferSize=530904
   [junit4]   2> 2382543 INFO  (qtp311316050-49077) [n:127.0.0.1:39603_xh_na%2Fct    x:collection1_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n2&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2383562 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 2383605 INFO  (qtp311316050-49077) [n:127.0.0.1:39603_xh_na%2Fct c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 2383632 INFO  (qtp311316050-49077) [n:127.0.0.1:39603_xh_na%2Fct c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema [collection1_shard1_replica_n2] Schema name=test
   [junit4]   2> 2383632 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=test
   [junit4]   2> 2383823 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2383830 INFO  (qtp311316050-49077) [n:127.0.0.1:39603_xh_na%2Fct c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2383847 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 2383849 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2383850 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-1-001/cores/collection1_shard2_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-1-001/cores/collection1_shard2_replica_n1/data/]
   [junit4]   2> 2383856 INFO  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1568650642, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2383858 WARN  (qtp2095660133-49049) [n:127.0.0.1:36331_xh_na%2Fct c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2383870 INFO  (qtp311316050-49077) [n:127.0.0.1:39603_xh_na%2Fct c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n2' using configuration from collection collection1, trusted=true
   [junit4]   2> 2383870 INFO  (qtp311316050-49077) [n:127.0.0.1:39603_xh_na%2Fct c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n2' (registry 'solr.core.collection1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c2a66
   [junit4]   2> 2383871 INFO  (qtp311316050-49077) [n:127.0.0.1:39603_xh_na%2Fct c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [[collection1_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-2-001/cores/collection1_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001/shard-2-001/cores/collection1_shard1_replica_n2/data/]
   [junit4]   2> 2383877 INFO  (qtp311316050-49077) [n:127.0.0.1:39603_xh_na%2Fct c:collection1 s:shard1  x:collection1_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1568650642, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2383879 WARN  (qtp311316050-49077) [n:127.

[...truncated too long message...]

ecting to 127.0.0.1 44374
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ForceLeaderTest -Dtests.method=testReplicasInLIRNoLeader -Dtests.seed=82FC50136F61DD87 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ar-IQ -Dtests.timezone=Asia/Taipei -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    153s J2 | ForceLeaderTest.testReplicasInLIRNoLeader <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:37517/xh_na/ct/forceleader_test_collection
   [junit4]    > at __randomizedtesting.SeedInfo.seed([82FC50136F61DD87:646B64D356E324E6]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
   [junit4]    > at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
   [junit4]    > at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:294)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]    > at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]    > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]    > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]    > at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > ... 52 more
   [junit4] IGNOR/A 0.00s J2 | ForceLeaderTest.testReplicasInLowerTerms
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_82FC50136F61DD87-001
   [junit4]   2> Jan 07, 2019 5:12:00 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-IQ, timezone=Asia/Taipei
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=109892120,total=535298048
   [junit4]   2> NOTE: All tests run in this JVM: [ByteBuffersDirectoryFactoryTest, ClusterStateUpdateTest, TestHdfsUpdateLog, DistributedUpdateProcessorTest, TestSchemaVersionResource, DebugComponentTest, AtomicUpdatesTest, AsyncCallRequestStatusResponseTest, TestLegacyNumericRangeQueryBuilder, ConnectionManagerTest, TestCustomSort, TriLevelCompositeIdRoutingTest, TestDocumentBuilder, SolrCloudReportersTest, TestSolrConfigHandlerCloud, TestStressVersions, AutoScalingHandlerTest, MigrateRouteKeyTest, DeleteInactiveReplicaTest, SolrCoreTest, TestTrie, AddReplicaTest, HdfsDirectoryFactoryTest, TestStressInPlaceUpdates, PreAnalyzedUpdateProcessorTest, TestOrdValues, TestMiniSolrCloudClusterSSL, TestLeaderElectionZkExpiry, TestChildDocTransformerHierarchy, TestInitParams, TestCodecSupport, TestHighFrequencyDictionaryFactory, TestCloudInspectUtil, TestPayloadCheckQParserPlugin, TestSimDistribStateManager, TestManagedStopFilterFactory, CustomCollectionTest, ScheduledTriggerIntegrationTest, JSONWriterTest, HdfsChaosMonkeyNothingIsSafeTest, BasicZkTest, BinaryUpdateRequestHandlerTest, NumericFieldsTest, SolrXmlInZkTest, TestSearcherReuse, BadCopyFieldTest, PeerSyncWithLeaderTest, TestPKIAuthenticationPlugin, TemplateUpdateProcessorTest, DistributedTermsComponentTest, SuggestComponentTest, TestCloudSchemaless, TestTolerantUpdateProcessorCloud, HDFSCollectionsAPITest, TestClusterStateMutator, TestConfigSetsAPIZkFailure, ReplaceNodeTest, TestCharFilters, HdfsRecoverLeaseTest, CoreAdminRequestStatusTest, AnalyticsMergeStrategyTest, HdfsAutoAddReplicasIntegrationTest, StatsReloadRaceTest, CoreSorterTest, XsltUpdateRequestHandlerTest, TestFieldCacheSortRandom, TestLegacyFieldCache, FullHLLTest, MultiThreadedOCPTest, SpellCheckCollatorWithCollapseTest, TestPostingsSolrHighlighter, SolrRequestParserTest, RollingRestartTest, ResponseLogComponentTest, TestCloudPseudoReturnFields, SignatureUpdateProcessorFactoryTest, DistributedQueueTest, ZkCLITest, DistributedIntervalFacetingTest, TimeRoutedAliasUpdateProcessorTest, TestSolrDeletionPolicy2, TestRandomRequestDistribution, TestCollectionAPIs, SuggesterTSTTest, TestComplexPhraseLeadingWildcard, SearchRateTriggerTest, DistanceUnitsTest, TestLegacyNumericUtils, TestPerFieldSimilarityWithDefaultOverride, MetricTriggerIntegrationTest, TestDynamicFieldCollectionResource, TestEmbeddedSolrServerAdminHandler, NodeMarkersRegistrationTest, HdfsLockFactoryTest, SimpleCollectionCreateDeleteTest, TestJoin, TestHdfsCloudBackupRestore, HttpPartitionTest, RandomizedTaggerTest, OrderedExecutorTest, RecoveryZkTest, TestReload, TestRestManager, SolrIndexConfigTest, ChangedSchemaMergeTest, TestJsonFacets, BJQParserTest, FileBasedSpellCheckerTest, SparseHLLTest, TestAuthenticationFramework, DocValuesMultiTest, CdcrRequestHandlerTest, FieldAnalysisRequestHandlerTest, LeaderVoteWaitTimeoutTest, HdfsUnloadDistributedZkTest, CleanupOldIndexTest, SolrCoreMetricManagerTest, TestIndexingPerformance, AlternateDirectoryTest, NodeAddedTriggerTest, TestSimNodeAddedTrigger, VersionInfoTest, BigEndianAscendingWordSerializerTest, TestHttpShardHandlerFactory, TestSSLRandomization, TestMergePolicyConfig, TestSimTriggerIntegration, ConvertedLegacyTest, TransactionLogTest, SolrTestCaseJ4Test, TestSort, DistributedFacetPivotSmallAdvancedTest, DistribCursorPagingTest, CdcrUpdateLogTest, ManagedSchemaRoundRobinCloudTest, CollectionStateFormat2Test, CollectionsAPISolrJTest, DeleteShardTest, ForceLeaderTest]
   [junit4] Completed [404/835 (1!)] on J2 in 153.71s, 3 tests, 1 error, 2 skipped <<< FAILURES!

[...truncated 10449 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:651: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid24753.hprof

Total time: 283 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 426 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/426/

2 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([1B45B21C0A901F9:89E064FB6E556C01]:0)
        at org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:596)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1089)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:36820/k_dpv/c/forceleader_test_collection

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:36820/k_dpv/c/forceleader_test_collection
        at __randomizedtesting.SeedInfo.seed([1B45B21C0A901F9:E7236FE1F92BF898]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:294)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 52 more




Build Log:
[...truncated 14576 lines...]
   [junit4] Suite: org.apache.solr.TestDistributedSearch
   [junit4]   2> 1822373 INFO  (SUITE-TestDistributedSearch-seed#[1B45B21C0A901F9]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/init-core-data-001
   [junit4]   2> 1822374 WARN  (SUITE-TestDistributedSearch-seed#[1B45B21C0A901F9]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=18 numCloses=18
   [junit4]   2> 1822375 INFO  (SUITE-TestDistributedSearch-seed#[1B45B21C0A901F9]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1822377 INFO  (SUITE-TestDistributedSearch-seed#[1B45B21C0A901F9]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-9061)
   [junit4]   2> 1822377 INFO  (SUITE-TestDistributedSearch-seed#[1B45B21C0A901F9]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1822620 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/tempDir-001/control/cores/collection1
   [junit4]   2> 1822622 WARN  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1822622 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1822622 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1822622 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1822623 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1822623 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1822623 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1822624 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@319d0afd{/,null,AVAILABLE}
   [junit4]   2> 1822626 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6c7807ab{HTTP/1.1,[http/1.1]}{127.0.0.1:40228}
   [junit4]   2> 1822626 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.Server Started @1822681ms
   [junit4]   2> 1822626 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=40228, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/tempDir-001/control/cores}
   [junit4]   2> 1822626 ERROR (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1822626 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1822626 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1822626 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1822626 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1822626 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-08T02:06:14.960Z
   [junit4]   2> 1822627 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/tempDir-001/control/solr.xml
   [junit4]   2> 1822643 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1822643 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1822651 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5beda6a0, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1822902 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1822902 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1822954 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5beda6a0
   [junit4]   2> 1822970 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5beda6a0
   [junit4]   2> 1822971 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5beda6a0
   [junit4]   2> 1822974 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/tempDir-001/control/cores
   [junit4]   2> 1822974 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1822975 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1823027 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1823364 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1823569 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1823588 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 1823591 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5beda6a0
   [junit4]   2> 1823592 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/tempDir-001/control/cores/collection1/data/]
   [junit4]   2> 1823609 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=44, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.3974609375, noCFSRatio=0.7126868047002407]
   [junit4]   2> 1823742 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1823742 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1823746 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1823746 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1823747 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.83203125, noCFSRatio=0.46339946905446816]
   [junit4]   2> 1823749 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4c45abce[collection1] main]
   [junit4]   2> 1823750 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1823763 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1823821 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1823821 INFO  (coreLoadExecutor-8153-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1622056030599905280
   [junit4]   2> 1823823 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1823823 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1823824 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1823824 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1823824 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1823824 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1823824 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1823825 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1823825 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1823826 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4c45abce[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1824087 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/tempDir-001/shard0/cores/collection1
   [junit4]   2> 1824088 WARN  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1824088 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1824088 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1824088 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1824097 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1824097 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1824097 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1824101 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@493c2aaa{/,null,AVAILABLE}
   [junit4]   2> 1824102 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5ac9545d{HTTP/1.1,[http/1.1]}{127.0.0.1:44740}
   [junit4]   2> 1824102 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.e.j.s.Server Started @1824158ms
   [junit4]   2> 1824102 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=44740, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/tempDir-001/shard0/cores}
   [junit4]   2> 1824103 ERROR (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1824103 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1824103 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1824103 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1824103 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1824103 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-08T02:06:16.437Z
   [junit4]   2> 1824103 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/tempDir-001/shard0/solr.xml
   [junit4]   2> 1824110 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1824110 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1824112 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5beda6a0, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1824377 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1824377 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1824412 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5beda6a0
   [junit4]   2> 1824433 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5beda6a0
   [junit4]   2> 1824433 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5beda6a0
   [junit4]   2> 1824437 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/tempDir-001/shard0/cores
   [junit4]   2> 1824437 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1824438 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1824600 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1824643 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1824856 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1824872 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 1824872 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5beda6a0
   [junit4]   2> 1824872 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_1B45B21C0A901F9-001/tempDir-001/shard0/cores/collection1/data/]
   [junit4]   2> 1824877 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=44, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.3974609375, noCFSRatio=0.7126868047002407]
   [junit4]   2> 1824989 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1824989 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1824992 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1824992 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1824995 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.83203125, noCFSRatio=0.46339946905446816]
   [junit4]   2> 1824997 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@37ae63b7[collection1] main]
   [junit4]   2> 1824998 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 1825012 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1825266 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1825266 INFO  (coreLoadExecutor-8167-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1622056032115097600
   [junit4]   2> 1825268 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 1825268 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 1825268 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 1825268 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 1825269 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 1825269 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 1825269 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 1825270 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 1825270 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 1825272 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@37ae63b7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1825282 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 1825290 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1622056032138166272)} 0 1
   [junit4]   2> 1825294 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1622056032143409152)} 0 1
   [junit4]   2> 1825301 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1 (1622056032145506304)]} 0 6
   [junit4]   2> 1825304 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1 (1622056032152846336)]} 0 2
   [junit4]   2> 1825306 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[2 (1622056032155992064)]} 0 0
   [junit4]   2> 1825307 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[2 (1622056032157040640)]} 0 0
   [junit4]   2> 1825308 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[3 (1622056032158089216)]} 0 0
   [junit4]   2> 1825309 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[3 (1622056032159137792)]} 0 0
   [junit4]   2> 1825314 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[4 (1622056032165429248)]} 0 0
   [junit4]   2> 1825325 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[4 (1622056032176963584)]} 0 0
   [junit4]   2> 1825329 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[5 (1622056032180109312)]} 0 0
   [junit4]   2> 1825330 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[5 (1622056032181157888)]} 0 0
   [junit4]   2> 1825330 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[6 (1622056032182206464)]} 0 0
   [junit4]   2> 1825331 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[6 (1622056032183255040)]} 0 0
   [junit4]   2> 1825332 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[7 (1622056032184303616)]} 0 0
   [junit4]   2> 1825333 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[7 (1622056032185352192)]} 0 0
   [junit4]   2> 1825336 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[8 (1622056032188497920)]} 0 0
   [junit4]   2> 1825337 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[8 (1622056032189546496)]} 0 0
   [junit4]   2> 1825338 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[9 (1622056032190595072)]} 0 0
   [junit4]   2> 1825339 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[9 (1622056032191643648)]} 0 0
   [junit4]   2> 1825340 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1622056032192692224,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1825340 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@22759cdb commitCommandVersion:1622056032192692224
   [junit4]   2> 1825383 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@419305c2[collection1] main]
   [junit4]   2> 1825383 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1825386 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@419305c2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.7.0):C9:[diagnostics={os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, os.arch=amd64, java.runtime.version=1.8.0_191-b12, source=flush, os.version=4.4.0-137-generic, timestamp=1546913177693}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
   [junit4]   2> 1825386 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 45
   [junit4]   2> 1825387 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1622056032241975296,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1825387 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@662048ee commitCommandVersion:1622056032241975296
   [junit4]   2> 1825443 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@49582064[collection1] main]
   [junit4]   2> 1825443 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1825446 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@49582064[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.7.0):C9:[diagnostics={os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, os.arch=amd64, java.runtime.version=1.8.0_191-b12, source=flush, os.version=4.4.0-137-generic, timestamp=1546913177742}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
   [junit4]   2> 1825446 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 59
   [junit4]   2> 1825450 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[10 (1622056032304889856)]} 0 3
   [junit4]   2> 1825454 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[10 (1622056032310132736)]} 0 2
   [junit4]   2> 1825460 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[11 (1622056032315375616)]} 0 2
   [junit4]   2> 1825462 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[11 (1622056032319569920)]} 0 1
   [junit4]   2> 1825464 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[12 (1622056032322715648)]} 0 0
   [junit4]   2> 1825465 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[12 (1622056032323764224)]} 0 0
   [junit4]   2> 1825466 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[13 (1622056032324812800)]} 0 0
   [junit4]   2> 1825467 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[13 (1622056032325861376)]} 0 0
   [junit4]   2> 1825469 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1001 (1622056032326909952)]} 0 0
   [junit4]   2> 1825471 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1001 (1622056032329007104)]} 0 0
   [junit4]   2> 1825472 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[14 (1622056032331104256)]} 0 0
   [junit4]   2> 1825476 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[14 (1622056032335298560)]} 0 0
   [junit4]   2> 1825477 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[15 (1622056032336347136)]} 0 0
   [junit4]   2> 1825478 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[15 (1622056032337395712)]} 0 0
   [junit4]   2> 1825481 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[16 (1622056032339492864)]} 0 0
   [junit4]   2> 1825482 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[16 (1622056032341590016)]} 0 0
   [junit4]   2> 1825483 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[17 (1622056032342638592)]} 0 0
   [junit4]   2> 1825487 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[17 (1622056032346832896)]} 0 0
   [junit4]   2> 1825492 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[100 (1622056032352075776)]} 0 0
   [junit4]   2> 1825493 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[100 (1622056032353124352)]} 0 0
   [junit4]   2> 1825494 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[101 (1622056032354172928)]} 0 0
   [junit4]   2> 1825496 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[101 (1622056032355221504)]} 0 0
   [junit4]   2> 1825496 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[102 (1622056032356270080)]} 0 0
   [junit4]   2> 1825498 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[102 (1622056032358367232)]} 0 0
   [junit4]   2> 1825499 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[103 (1622056032359415808)]} 0 0
   [junit4]   2> 1825501 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[103 (1622056032361512960)]} 0 0
   [junit4]   2> 1825502 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[104 (1622056032362561536)]} 0 0
   [junit4]   2> 1825504 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[104 (1622056032364658688)]} 0 0
   [junit4]   2> 1825505 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[105 (1622056032365707264)]} 0 0
   [junit4]   2> 1825505 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[105 (1622056032365707264)]} 0 0
   [junit4]   2> 1825507 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[106 (1622056032367804416)]} 0 0
   [junit4]   2> 1825508 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[106 (1622056032367804416)]} 0 0
   [junit4]   2> 1825509 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[107 (1622056032369901568)]} 0 0
   [junit4]   2> 1825509 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[107 (1622056032369901568)]} 0 0
   [junit4]   2> 1825510 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[108 (1622056032370950144)]} 0 0
   [junit4]   2> 1825512 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[108 (1622056032373047296)]} 0 0
   [junit4]   2> 1825513 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[109 (1622056032374095872)]} 0 0
   [junit4]   2> 1825514 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[109 (1622056032375144448)]} 0 0
   [junit4]   2> 1825515 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[110 (1622056032376193024)]} 0 0
   [junit4]   2> 1825516 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[110 (1622056032377241600)]} 0 0
   [junit4]   2> 1825517 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[111 (1622056032377241600)]} 0 0
   [junit4]   2> 1825517 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[111 (1622056032378290176)]} 0 0
   [junit4]   2> 1825520 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[112 (1622056032380387328)]} 0 0
   [junit4]   2> 1825521 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[112 (1622056032381435904)]} 0 0
   [junit4]   2> 1825521 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[113 (1622056032382484480)]} 0 0
   [junit4]   2> 1825523 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[113 (1622056032384581632)]} 0 0
   [junit4]   2> 1825523 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[114 (1622056032384581632)]} 0 0
   [junit4]   2> 1825524 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[114 (1622056032385630208)]} 0 0
   [junit4]   2> 1825525 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[115 (1622056032386678784)]} 0 0
   [junit4]   2> 1825526 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[115 (1622056032386678784)]} 0 0
   [junit4]   2> 1825526 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[116 (1622056032387727360)]} 0 0
   [junit4]   2> 1825527 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[116 (1622056032388775936)]} 0 0
   [junit4]   2> 1825528 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[117 (1622056032389824512)]} 0 0
   [junit4]   2> 1825529 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[117 (1622056032390873088)]} 0 0
   [junit4]   2> 1825529 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[118 (1622056032390873088)]} 0 0
   [junit4]   2> 1825530 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[118 (1622056032391921664)]} 0 0
   [junit4]   2> 1825531 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[119 (1622056032392970240)]} 0 0
   [junit4]   2> 1825534 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[119 (1622056032396115968)]} 0 0
   [junit4]   2> 1825535 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[120 (1622056032396115968)]} 0 0
   [junit4]   2> 1825535 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[120 (1622056032397164544)]} 0 0
   [junit4]   2> 1825536 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[121 (1622056032398213120)]} 0 0
   [junit4]   2> 1825537 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[121 (1622056032399261696)]} 0 0
   [junit4]   2> 1825539 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[122 (1622056032401358848)]} 0 0
   [junit4]   2> 1825540 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[122 (1622056032402407424)]} 0 0
   [junit4]   2> 1825541 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[123 (1622056032403456000)]} 0 0
   [junit4]   2> 1825542 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[123 (1622056032403456000)]} 0 0
   [junit4]   2> 1825543 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[124 (1622056032404504576)]} 0 0
   [junit4]   2> 1825545 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[124 (1622056032407650304)]} 0 0
   [junit4]   2> 1825546 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[125 (1622056032408698880)]} 0 0
   [junit4]   2> 1825547 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[125 (1622056032408698880)]} 0 0
   [junit4]   2> 1825548 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[126 (1622056032410796032)]} 0 0
   [junit4]   2> 1825550 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[126 (1622056032412893184)]} 0 0
   [junit4]   2> 1825551 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[127 (1622056032413941760)]} 0 0
   [junit4]   2> 1825552 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[127 (1622056032414990336)]} 0 0
   [junit4]   2> 1825553 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[128 (1622056032416038912)]} 0 0
   [junit4]   2> 1825558 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[128 (1622056032421281792)]} 0 0
   [junit4]   2> 1825559 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[129 (1622056032422330368)]} 0 0
   [junit4]   2> 1825560 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[129 (1622056032423378944)]} 0 0
   [junit4]   2> 1825561 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[130 (1622056032424427520)]} 0 0
   [junit4]   2> 1825561 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[130 (1622056032424427520)]} 0 0
   [junit4]   2> 1825562 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[131 (1622056032425476096)]} 0 0
   [junit4]   2> 1825563 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[131 (1622056032426524672)]} 0 0
   [junit4]   2> 1825568 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[132 (1622056032431767552)]} 0 0
   [junit4]   2> 1825569 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[132 (1622056032432816128)]} 0 0
   [junit4]   2> 1825570 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[133 (1622056032433864704)]} 0 0
   [junit4]   2> 1825571 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[133 (1622056032434913280)]} 0 0
   [junit4]   2> 1825573 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[134 (1622056032435961856)]} 0 0
   [junit4]   2> 1825573 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[134 (1622056032437010432)]} 0 0
   [junit4]   2> 1825574 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[135 (1622056032438059008)]} 0 0
   [junit4]   2> 1825575 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[135 (1622056032438059008)]} 0 0
   [junit4]   2> 1825575 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[136 (1622056032439107584)]} 0 0
   [junit4]   2> 1825576 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[136 (1622056032440156160)]} 0 0
   [junit4]   2> 1825578 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[137 (1622056032441204736)]} 0 0
   [junit4]   2> 1825579 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[137 (1622056032443301888)]} 0 0
   [junit4]   2> 1825580 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[138 (1622056032444350464)]} 0 0
   [junit4]   2> 1825580 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[138 (1622056032444350464)]} 0 0
   [junit4]   2> 1825581 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[139 (1622056032445399040)]} 0 0
   [junit4]   2> 1825582 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[139 (1622056032446447616)]} 0 0
   [junit4]   2> 1825583 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[140 (1622056032447496192)]} 0 0
   [junit4]   2> 1825584 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[140 (1622056032448544768)]} 0 0
   [junit4]   2> 1825585 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[141 (1622056032448544768)]} 0 0
   [junit4]   2> 1825586 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[141 (1622056032450641920)]} 0 0
   [junit4]   2> 1825586 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[142 (1622056032450641920)]} 0 0
   [junit4]   2> 1825587 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[142 (1622056032451690496)]} 0 0
   [junit4]   2> 1825588 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[143 (1622056032451690496)]} 0 0
   [junit4]   2> 1825588 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[143 (1622056032452739072)]} 0 0
   [junit4]   2> 1825591 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[144 (1622056032455884800)]} 0 0
   [junit4]   2> 1825592 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[144 (1622056032456933376)]} 0 0
   [junit4]   2> 1825593 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[145 (1622056032456933376)]} 0 0
   [junit4]   2> 1825593 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[145 (1622056032457981952)]} 0 0
   [junit4]   2> 1825594 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[146 (1622056032459030528)]} 0 0
   [junit4]   2> 1825594 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[146 (1622056032459030528)]} 0 0
   [junit4]   2> 1825595 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[147 (1622056032460079104)]} 0 0
   [junit4]   2> 1825596 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[147 (1622056032460079104)]} 0 0
   [junit4]   2> 1825596 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[148 (1622056032461127680)]} 0 0
   [junit4]   2> 1825599 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[148 (1622056032464273408)]} 0 0
   [junit4]   2> 1825600 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[149 (1622056032465321984)]} 0 0
   [junit4]   2> 1825602 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[149 (1622056032467419136)]} 0 0
   [junit4]   2> 1825603 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1622056032468467712,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1825603 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@22759cdb commitCommandVersion:1622056032468467712
   [junit4]   2> 1825667 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@15ad5193[collection1] main]
   [junit4]   2> 1825667 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1825670 INFO  (searcherExecutor-8154-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@15ad5193[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.7.0):C9:[diagnostics={os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, os.arch=amd64, java.runtime.version=1.8.0_191-b12, source=flush, os.version=4.4.0-137-generic, timestamp=1546913177693}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_1(7.7.0):C59:[diagnostics={os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, os.arch=amd64, java.runtime.version=1.8.0_191-b12, source=flush, os.version=4.4.0-137-generic, timestamp=1546913177983}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
   [junit4]   2> 1825671 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 68
   [junit4]   2> 1825672 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1622056032540819456,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1825672 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@662048ee commitCommandVersion:1622056032540819456
   [junit4]   2> 1825726 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@673ff531[collection1] main]
   [junit4]   2> 1825727 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1825731 INFO  (searcherExecutor-8168-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@673ff531[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.7.0):C9:[diagnostics={os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, os.arch=amd64, java.runtime.version=1.8.0_191-b12, source=flush, os.version=4.4.0-137-generic, timestamp=1546913177742}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_1(7.7.0):C59:[diagnostics={os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_191, java.vm.version=25.191-b12, lucene.version=7.7.0, os.arch=amd64, java.runtime.version=1.8.0_191-b12, source=flush, os.version=4.4.0-137-generic, timestamp=1546913178032}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
   [junit4]   2> 1825731 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 59
   [junit4]   2> 1825732 ERROR (qtp175360178-40788) [    x:collection1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: For input string: "non_numeric_value"
   [junit4]   2> at org.apache.solr.common.params.SolrParams.getInt(SolrParams.java:236)
   [junit4]   2> at org.apache.solr.search.QParser.getSortSpec(QParser.java:270)
   [junit4]   2> at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:187)
   [junit4]   2> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:272)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:502)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NumberFormatException: For input string: "non_numeric_value"
   [junit4]   2> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
   [junit4]   2> at java.lang.Integer.parseInt(Integer.java:580)
   [junit4]   2> at java.lang.Integer.valueOf(Integer.java:766)
   [junit4]   2> at org.apache.solr.common.params.SolrParams.getInt(SolrParams.java:233)
   [junit4]   2> ... 34 more
   [junit4]   2>
   [junit4]   2> 1825732 INFO  (qtp175360178-40788) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*&distrib=false&start=non_numeric_value&wt=javabin&version=2} status=400 QTime=0
   [junit4]   2> 1825734 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.h.RequestHandlerBase Ignoring exception matching parameter cannot be negative
   [junit4]   2> 1825734 INFO  (qtp175360178-40789) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*&distrib=false&start=-1&wt=javabin&version=2} status=400 QTime=0
   [junit4]   2> 1825735 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.h.RequestHandlerBase Ignoring exception matching parameter cannot be negative
   [junit4]   2> 1825735 INFO  (qtp175360178-40790) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*&distrib=false&start=0&rows=-1&wt=javabin&version=2} status=400 QTime=0
   [junit4]   2> 1825736 ERROR (qtp175360178-40786) [    x:collection1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: For input string: "non_numeric_value"
   [junit4]   2> at org.apache.solr.common.params.SolrParams.getInt(SolrParams.java:236)
   [junit4]   2> at org.apache.solr.search.QParser.getSortSpec(QParser.java:273)
   [junit4]   2> at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:187)
   [junit4]   2> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:272)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:502)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NumberFormatException: For input string: "non_numeric_value"
   [junit4]   2> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
   [junit4]   2> at java.lang.Integer.parseInt(Integer.java:580)
   [junit4]   2> at java.lang.Integer.valueOf(Integer.java:766)
   [junit4]   2> at org.apache.solr.common.params.SolrParams.getInt(SolrParams.java:233)
   [junit4]   2> ... 34 more
   [junit4]   2>
   [junit4]   2> 1825736 INFO  (qtp175360178-40786) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*&distrib=false&rows=non_numeric_value&wt=javabin&version=2} status=400 QTime=0
   [junit4]   2> 1825737 INFO  (qtp175360178-40787) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&sort=n_ti1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
   [junit4]   2> 1825743 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+desc&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&rows=10&version=2&q=*:*&NOW=1546913178073&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1825772 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1546913178073&ids=134,3,135,116,105,1001,7,119,141,131&isShard=true&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1825774 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&sort=n_ti1+desc&wt=javabin&version=2} hits=68 status=0 QTime=34
   [junit4]   2> 1825774 INFO  (TEST-TestDistributedSearch.test-seed#[1B45B21C0A901F9]) [    ] o.a.s.BaseDistributedSearchTestCase starting stress...
   [junit4]   2> 1825791 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+desc&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&rows=10&version=2&q=*:*&NOW=1546913178117&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1825792 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1546913178117&ids=134,3,135,116,105,1001,7,119,141,131&isShard=true&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1825794 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&sort=n_ti1+desc&wt=javabin&version=2} hits=68 status=0 QTime=10
   [junit4]   2> 1825798 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+desc&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&rows=10&version=2&q=*:*&NOW=1546913178128&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1825806 INFO  (qtp1647359392-40808) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1546913178128&ids=134,3,135,116,105,1001,7,119,141,131&isShard=true&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1825806 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+desc&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&rows=10&version=2&q=*:*&NOW=1546913178132&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1825807 INFO  (qtp1647359392-40810) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+desc&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&rows=10&version=2&q=*:*&NOW=1546913178131&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1825807 INFO  (qtp1647359392-40809) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&sort=n_ti1+desc&wt=javabin&version=2} hits=68 status=0 QTime=12
   [junit4]   2> 1825809 INFO  (qtp1647359392-40833) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1546913178132&ids=134,3,135,116,105,1001,7,119,141,131&isShard=true&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1825810 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&sort=n_ti1+desc&wt=javabin&version=2} hits=68 status=0 QTime=12
   [junit4]   2> 1825814 INFO  (qtp1647359392-40834) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1546913178131&ids=134,3,135,116,105,1001,7,119,141,131&isShard=true&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1825815 INFO  (qtp1647359392-40806) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&shards=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&sort=n_ti1+desc&wt=javabin&version=2} hits=68 status=0 QTime=17
   [junit4]   2> 1825815 INFO  (qtp1647359392-40834) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+desc&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&rows=10&version=2&q=*:*&NOW=1546913178144&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1825816 INFO  (qtp1647359392-40807) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+desc&shard.url=http://127.0.0.1:44740//collection1|[ff01::083]:33332/|[ff01::114]:33332/|[ff01::213]:33332/&rows=10&version=2&q=*:*&NOW=1546913178150&isShard=true&wt=javabin} hits=68 status=0 QTime=0
   [junit4]   2> 1825816 INFO  (qtp1647359392-40835) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={

[...truncated too long message...]

vy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1477367575
     [copy] Copying 242 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1477367575
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1477367575

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 49ms :: artifacts dl 1ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 302 minutes 44 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 427 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/427/

2 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=158825, name=testExecutor-14983-thread-17, state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=158825, name=testExecutor-14983-thread-17, state=RUNNABLE, group=TGRP-HdfsUnloadDistributedZkTest]
        at __randomizedtesting.SeedInfo.seed([6D27CF39EEEFD016:E573F0E34013BDEE]:0)
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:36518
        at __randomizedtesting.SeedInfo.seed([6D27CF39EEEFD016]:0)
        at org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCollectionInOneInstance$1(BasicDistributedZkTest.java:659)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:36518
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCollectionInOneInstance$1(BasicDistributedZkTest.java:657)
        ... 4 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 9 more


FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:38732/forceleader_test_collection

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:38732/forceleader_test_collection
        at __randomizedtesting.SeedInfo.seed([6D27CF39EEEFD016:8BB0FBF9D76D2977]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:294)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 52 more




Build Log:
[...truncated 14949 lines...]
   [junit4] Suite: org.apache.solr.cloud.ForceLeaderTest
   [junit4]   2> 4244752 INFO  (SUITE-ForceLeaderTest-seed#[6D27CF39EEEFD016]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/init-core-data-001
   [junit4]   2> 4244773 WARN  (SUITE-ForceLeaderTest-seed#[6D27CF39EEEFD016]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=105 numCloses=105
   [junit4]   2> 4244773 INFO  (SUITE-ForceLeaderTest-seed#[6D27CF39EEEFD016]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 4244774 INFO  (SUITE-ForceLeaderTest-seed#[6D27CF39EEEFD016]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 4244774 INFO  (SUITE-ForceLeaderTest-seed#[6D27CF39EEEFD016]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 4244883 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4244890 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4244890 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4244995 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer start zk server on port:35698
   [junit4]   2> 4244995 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:35698
   [junit4]   2> 4244995 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 35698
   [junit4]   2> 4245076 INFO  (zkConnectionManagerCallback-7611-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4245121 INFO  (zkConnectionManagerCallback-7613-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4245122 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 4245123 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 4245124 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 4245124 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 4245142 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 4245142 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 4245143 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 4245143 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 4245144 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 4245145 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 4245158 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 4245159 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 4245589 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4245589 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4245589 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4245590 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4245600 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4245600 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4245600 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4245600 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@35f28c86{/,null,AVAILABLE}
   [junit4]   2> 4245601 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@70f55e9e{HTTP/1.1,[http/1.1]}{127.0.0.1:42720}
   [junit4]   2> 4245601 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.e.j.s.Server Started @4245648ms
   [junit4]   2> 4245601 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=36216, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/control-001/cores}
   [junit4]   2> 4245601 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4245601 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4245601 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 4245601 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4245601 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4245601 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-09T11:21:51.923Z
   [junit4]   2> 4245642 INFO  (zkConnectionManagerCallback-7615-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4245643 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4245643 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/control-001/solr.xml
   [junit4]   2> 4245646 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4245646 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4245648 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4246032 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35698/solr
   [junit4]   2> 4246034 INFO  (zkConnectionManagerCallback-7619-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4246035 INFO  (zkConnectionManagerCallback-7621-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4246383 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [n:127.0.0.1:36216_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36216_
   [junit4]   2> 4246384 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer Overseer (id=73982406814990340-127.0.0.1:36216_-n_0000000000) starting
   [junit4]   2> 4246421 INFO  (zkConnectionManagerCallback-7628-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4246422 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [n:127.0.0.1:36216_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35698/solr ready
   [junit4]   2> 4246438 INFO  (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:36216_
   [junit4]   2> 4246438 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [n:127.0.0.1:36216_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36216_
   [junit4]   2> 4246439 INFO  (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4246458 INFO  (zkCallback-7627-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4246535 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [n:127.0.0.1:36216_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4246595 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [n:127.0.0.1:36216_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4246632 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [n:127.0.0.1:36216_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4246640 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [n:127.0.0.1:36216_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4246641 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [n:127.0.0.1:36216_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/control-001/cores
   [junit4]   2> 4246766 INFO  (zkConnectionManagerCallback-7634-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4246767 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4246768 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35698/solr ready
   [junit4]   2> 4246769 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=54154,localport=36216], receiveBufferSize:531000
   [junit4]   2> 4246772 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=42720,localport=56356], receiveBufferSize=530904
   [junit4]   2> 4246786 INFO  (qtp704614131-36263) [n:127.0.0.1:36216_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:36216_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4246788 INFO  (OverseerThreadFactory-7852-thread-1-processing-n:127.0.0.1:36216_) [n:127.0.0.1:36216_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 4246805 DEBUG (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"control_collection",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "createNodeSet":"127.0.0.1:36216_",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 4246893 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=54160,localport=36216], receiveBufferSize:531000
   [junit4]   2> 4246893 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=42720,localport=56362], receiveBufferSize=530904
   [junit4]   2> 4246895 INFO  (qtp704614131-36265) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4246908 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4246909 INFO  (qtp704614131-36262) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4246909 INFO  (qtp704614131-36264) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4246910 INFO  (qtp704614131-36265) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4246921 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4246921 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4246953 DEBUG (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:36216",
   [junit4]   2>   "node_name":"127.0.0.1:36216_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 4247986 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 4248009 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 4248275 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4248304 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 4248317 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4248317 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 4248319 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=51.2041015625, floorSegmentMB=0.515625, forceMergeDeletesPctAllowed=8.061552154829513, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6848966704227561, deletesPctAllowed=22.739297103763086
   [junit4]   2> 4248321 WARN  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4248535 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4248535 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4248558 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4248558 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4248559 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2124451713, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 4248559 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@426f1f64[control_collection_shard1_replica_n1] main]
   [junit4]   2> 4248593 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 4248593 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 4248593 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4248594 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1622181585790959616
   [junit4]   2> 4248597 INFO  (searcherExecutor-7857-thread-1-processing-n:127.0.0.1:36216_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@426f1f64[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4248647 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 4248647 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 4248662 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 4248662 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 4248662 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:36216/control_collection_shard1_replica_n1/
   [junit4]   2> 4248662 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 4248662 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:36216/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 4248662 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 4248663 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/73982406814990340-core_node2-n_0000000000
   [junit4]   2> 4248664 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:36216/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 4248664 DEBUG (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:36216",
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "state":"active"} current state version: 0
   [junit4]   2> 4248872 INFO  (zkCallback-7620-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4248872 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 4248874 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1953
   [junit4]   2> 4248884 INFO  (zkCallback-7620-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4248887 DEBUG (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "core_node_name":"core_node2",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:36216",
   [junit4]   2>   "node_name":"127.0.0.1:36216_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 4248917 INFO  (qtp704614131-36263) [n:127.0.0.1:36216_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 4248990 INFO  (zkCallback-7620-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4248990 INFO  (zkCallback-7620-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4248990 INFO  (zkCallback-7620-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4248991 INFO  (qtp704614131-36263) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:36216_&wt=javabin&version=2} status=0 QTime=2204
   [junit4]   2> 4249070 INFO  (zkConnectionManagerCallback-7639-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4249071 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4249072 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35698/solr ready
   [junit4]   2> 4249072 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 4249072 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=54176,localport=36216], receiveBufferSize:531000
   [junit4]   2> 4249073 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=42720,localport=56378], receiveBufferSize=530904
   [junit4]   2> 4249073 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4249084 INFO  (OverseerThreadFactory-7852-thread-2-processing-n:127.0.0.1:36216_) [n:127.0.0.1:36216_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 4249084 INFO  (OverseerCollectionConfigSetProcessor-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4249118 DEBUG (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "createNodeSet":"",
   [junit4]   2>   "stateFormat":"1",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 4249318 WARN  (OverseerThreadFactory-7852-thread-2-processing-n:127.0.0.1:36216_) [n:127.0.0.1:36216_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 4249320 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 4249321 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=247
   [junit4]   2> 4249322 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=3
   [junit4]   2> 4249839 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-1-001 of type NRT
   [junit4]   2> 4249852 WARN  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4249853 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4249853 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4249853 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4249929 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4249929 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4249929 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 4249947 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5a3b6e19{/,null,AVAILABLE}
   [junit4]   2> 4249962 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5405f46c{HTTP/1.1,[http/1.1]}{127.0.0.1:36896}
   [junit4]   2> 4249973 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.Server Started @4250009ms
   [junit4]   2> 4249974 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=41361, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-1-001/cores}
   [junit4]   2> 4249974 ERROR (closeThreadPool-7640-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4249974 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4249986 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 4249986 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4249986 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4249986 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-09T11:21:56.308Z
   [junit4]   2> 4250119 INFO  (zkConnectionManagerCallback-7642-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4250128 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4250128 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-1-001/solr.xml
   [junit4]   2> 4250144 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4250144 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4250145 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4250489 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35698/solr
   [junit4]   2> 4250505 INFO  (zkConnectionManagerCallback-7646-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4250545 INFO  (zkConnectionManagerCallback-7648-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4250564 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4250579 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.c.ZkController Publish node=127.0.0.1:41361_ as DOWN
   [junit4]   2> 4250579 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4250579 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41361_
   [junit4]   2> 4250581 INFO  (zkCallback-7627-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4250581 INFO  (zkCallback-7620-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4250593 INFO  (zkCallback-7638-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4250638 INFO  (zkCallback-7647-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 4250638 DEBUG (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:41361_"} current state version: 1
   [junit4]   2> 4250740 INFO  (zkConnectionManagerCallback-7655-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4250741 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 4250741 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35698/solr ready
   [junit4]   2> 4250784 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4250880 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4250930 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4250930 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4250931 INFO  (closeThreadPool-7640-thread-1) [n:127.0.0.1:41361_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-1-001/cores
   [junit4]   2> 4251108 INFO  (OverseerCollectionConfigSetProcessor-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4251259 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-2-001 of type NRT
   [junit4]   2> 4251260 WARN  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4251260 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4251260 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4251260 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4251261 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4251277 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4251277 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 4251310 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3116501a{/,null,AVAILABLE}
   [junit4]   2> 4251311 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@790114a2{HTTP/1.1,[http/1.1]}{127.0.0.1:37900}
   [junit4]   2> 4251311 INFO  (closeThreadPool-7640-thread-1) [    ] o.e.j.s.Server Started @4251358ms
   [junit4]   2> 4251311 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=41041, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-2-001/cores}
   [junit4]   2> 4251311 ERROR (closeThreadPool-7640-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4251311 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4251311 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 4251311 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4251311 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4251312 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-09T11:21:57.633Z
   [junit4]   2> 4251313 INFO  (zkConnectionManagerCallback-7658-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4251332 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4251332 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-2-001/solr.xml
   [junit4]   2> 4251349 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4251349 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4251363 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4252178 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35698/solr
   [junit4]   2> 4252194 INFO  (zkConnectionManagerCallback-7662-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4252285 INFO  (zkConnectionManagerCallback-7664-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4252344 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 4252347 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.ZkController Publish node=127.0.0.1:41041_ as DOWN
   [junit4]   2> 4252361 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4252361 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41041_
   [junit4]   2> 4252361 DEBUG (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:41041_"} current state version: 1
   [junit4]   2> 4252362 INFO  (zkCallback-7627-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4252362 INFO  (zkCallback-7620-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4252368 INFO  (zkCallback-7638-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4252384 INFO  (zkCallback-7647-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4252384 INFO  (zkCallback-7654-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4252413 INFO  (zkCallback-7663-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4252562 INFO  (zkConnectionManagerCallback-7671-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4252595 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4252595 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35698/solr ready
   [junit4]   2> 4252626 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[6D27CF39EEEFD016]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-3-001 of type NRT
   [junit4]   2> 4252631 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4252651 WARN  (closeThreadPool-7640-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 4252651 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 4252652 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 4252652 INFO  (closeThreadPool-7640-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 4252691 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4252713 INFO  (closeThreadPool-7640-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4252713 INFO  (closeThreadPool-7640-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4252713 INFO  (closeThreadPool-7640-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 4252714 INFO  (closeThreadPool-7640-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7d6a5652{/,null,AVAILABLE}
   [junit4]   2> 4252714 INFO  (closeThreadPool-7640-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@60d505ef{HTTP/1.1,[http/1.1]}{127.0.0.1:38938}
   [junit4]   2> 4252714 INFO  (closeThreadPool-7640-thread-2) [    ] o.e.j.s.Server Started @4252762ms
   [junit4]   2> 4252714 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=38732, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-3-001/cores}
   [junit4]   2> 4252715 ERROR (closeThreadPool-7640-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4252715 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4252715 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.0
   [junit4]   2> 4252715 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4252715 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4252715 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-09T11:21:59.037Z
   [junit4]   2> 4252741 INFO  (zkConnectionManagerCallback-7674-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4252742 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4252742 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-3-001/solr.xml
   [junit4]   2> 4252762 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4252762 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4252764 INFO  (closeThreadPool-7640-thread-1) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-2-001/cores
   [junit4]   2> 4252778 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4252778 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4252779 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4253332 INFO  (closeThreadPool-7640-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35698/solr
   [junit4]   2> 4253333 INFO  (zkConnectionManagerCallback-7678-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4253335 INFO  (zkConnectionManagerCallback-7680-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4253355 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4253358 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.c.ZkController Publish node=127.0.0.1:38732_ as DOWN
   [junit4]   2> 4253359 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4253359 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38732_
   [junit4]   2> 4253359 DEBUG (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:38732_"} current state version: 1
   [junit4]   2> 4253360 INFO  (zkCallback-7638-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4253360 INFO  (zkCallback-7627-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4253360 INFO  (zkCallback-7620-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4253360 INFO  (zkCallback-7663-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4253360 INFO  (zkCallback-7654-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4253368 INFO  (zkCallback-7670-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4253368 INFO  (zkCallback-7647-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4253368 INFO  (zkCallback-7679-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4253443 INFO  (zkConnectionManagerCallback-7687-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 4253444 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4253474 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35698/solr ready
   [junit4]   2> 4253489 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4253537 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4253562 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4253562 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4253576 INFO  (closeThreadPool-7640-thread-2) [n:127.0.0.1:38732_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-3-001/cores
   [junit4]   2> 4253675 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=54260,localport=36216], receiveBufferSize:531000
   [junit4]   2> 4253675 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=42720,localport=56464], receiveBufferSize=530904
   [junit4]   2> 4253675 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=54262,localport=36216], receiveBufferSize:531000
   [junit4]   2> 4253676 INFO  (qtp704614131-36262) [n:127.0.0.1:36216_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:41041_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4253676 INFO  (qtp704614131-36264) [n:127.0.0.1:36216_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:41361_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4253681 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=42720,localport=56466], receiveBufferSize=530904
   [junit4]   2> 4253764 INFO  (qtp704614131-36261) [n:127.0.0.1:36216_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:38732_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4253765 INFO  (qtp704614131-36263) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=81
   [junit4]   2> 4253766 INFO  (qtp704614131-36265) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 4253767 INFO  (qtp704614131-36263) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 4253768 INFO  (qtp704614131-36265) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253790 INFO  (qtp704614131-36263) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=21
   [junit4]   2> 4253791 INFO  (qtp704614131-36263) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253791 INFO  (qtp704614131-36263) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253792 INFO  (qtp704614131-36263) [n:127.0.0.1:36216_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253793 INFO  (SocketProxy-Acceptor-38732) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=46456,localport=38732], receiveBufferSize:531000
   [junit4]   2> 4253798 INFO  (SocketProxy-Acceptor-38732) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=38938,localport=36392], receiveBufferSize=530904
   [junit4]   2> 4253807 INFO  (qtp464547939-36392) [n:127.0.0.1:38732_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253809 INFO  (qtp464547939-36393) [n:127.0.0.1:38732_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253818 INFO  (qtp464547939-36394) [n:127.0.0.1:38732_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=8
   [junit4]   2> 4253819 INFO  (qtp464547939-36395) [n:127.0.0.1:38732_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253819 INFO  (qtp464547939-36391) [n:127.0.0.1:38732_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253820 INFO  (SocketProxy-Acceptor-41041) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56834,localport=41041], receiveBufferSize:531000
   [junit4]   2> 4253820 INFO  (SocketProxy-Acceptor-41041) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37900,localport=49018], receiveBufferSize=530904
   [junit4]   2> 4253821 INFO  (qtp2037187744-36362) [n:127.0.0.1:41041_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253822 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253822 INFO  (qtp2037187744-36364) [n:127.0.0.1:41041_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253823 INFO  (qtp2037187744-36365) [n:127.0.0.1:41041_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253824 INFO  (qtp2037187744-36361) [n:127.0.0.1:41041_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253824 INFO  (SocketProxy-Acceptor-41361) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=42236,localport=41361], receiveBufferSize:531000
   [junit4]   2> 4253824 INFO  (SocketProxy-Acceptor-41361) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36896,localport=52880], receiveBufferSize=530904
   [junit4]   2> 4253825 INFO  (qtp696400066-36329) [n:127.0.0.1:41361_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253838 INFO  (qtp696400066-36330) [n:127.0.0.1:41361_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=12
   [junit4]   2> 4253839 INFO  (qtp696400066-36332) [n:127.0.0.1:41361_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253840 INFO  (qtp696400066-36331) [n:127.0.0.1:41361_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253840 INFO  (qtp696400066-36328) [n:127.0.0.1:41361_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 4253841 INFO  (OverseerThreadFactory-7852-thread-3-processing-n:127.0.0.1:36216_) [n:127.0.0.1:36216_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:41041_ for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 4253841 INFO  (OverseerThreadFactory-7852-thread-4-processing-n:127.0.0.1:36216_) [n:127.0.0.1:36216_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:41361_ for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 4253842 INFO  (OverseerThreadFactory-7852-thread-3-processing-n:127.0.0.1:36216_) [n:127.0.0.1:36216_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 4253842 INFO  (OverseerThreadFactory-7852-thread-4-processing-n:127.0.0.1:36216_) [n:127.0.0.1:36216_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 4253843 INFO  (SocketProxy-Acceptor-41041) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56842,localport=41041], receiveBufferSize:531000
   [junit4]   2> 4253843 INFO  (SocketProxy-Acceptor-41361) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=42244,localport=41361], receiveBufferSize:531000
   [junit4]   2> 4253844 INFO  (qtp696400066-36330) [n:127.0.0.1:41361_    x:collection1_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n2&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4253844 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_    x:collection1_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n1&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 4253844 INFO  (SocketProxy-Acceptor-41041) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37900,localport=49026], receiveBufferSize=530904
   [junit4]   2> 4253845 INFO  (SocketProxy-Acceptor-41361) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36896,localport=52888], receiveBufferSize=530904
   [junit4]   2> 4253846 DEBUG (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
   [junit4]   2>   "core":"collection1_shard1_replica_n1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41041",
   [junit4]   2>   "node_name":"127.0.0.1:41041_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 1
   [junit4]   2> 4253860 DEBUG (OverseerStateUpdate-73982406814990340-127.0.0.1:36216_-n_0000000000) [n:127.0.0.1:36216_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1_shard2_replica_n2",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41361",
   [junit4]   2>   "node_name":"127.0.0.1:41361_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 4254905 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 4254916 INFO  (qtp696400066-36330) [n:127.0.0.1:41361_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 4254962 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=test
   [junit4]   2> 4254988 INFO  (qtp696400066-36330) [n:127.0.0.1:41361_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.s.IndexSchema [collection1_shard2_replica_n2] Schema name=test
   [junit4]   2> 4255720 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4255736 INFO  (qtp696400066-36330) [n:127.0.0.1:41361_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4255808 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 4255809 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n1' (registry 'solr.core.collection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4255810 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [[collection1_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-2-001/cores/collection1_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-2-001/cores/collection1_shard1_replica_n1/data/]
   [junit4]   2> 4255812 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=51.2041015625, floorSegmentMB=0.515625, forceMergeDeletesPctAllowed=8.061552154829513, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6848966704227561, deletesPctAllowed=22.739297103763086
   [junit4]   2> 4255814 WARN  (qtp2037187744-36363) [n:127.0.0.1:41041_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4255908 INFO  (qtp696400066-36330) [n:127.0.0.1:41361_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n2' using configuration from collection collection1, trusted=true
   [junit4]   2> 4255910 INFO  (qtp696400066-36330) [n:127.0.0.1:41361_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n2' (registry 'solr.core.collection1.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@16f178ff
   [junit4]   2> 4255910 INFO  (qtp696400066-36330) [n:127.0.0.1:41361_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.c.SolrCore [[collection1_shard2_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-1-001/cores/collection1_shard2_replica_n2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_6D27CF39EEEFD016-001/shard-1-001/cores/collection1_shard2_replica_n2/data/]
   [junit4]   2> 4255945 INFO  (qtp696400066-36330) [n:127.0.0.1:41361_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=51.2041015625, floorSegmentMB=0.515625, forceMergeDeletesPctAllowed=8.061552154829513, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6848966704227561, deletesPctAllowed=22.739297103763086
   [junit4]   2> 4255947 WARN  (qtp696400066-36330) [n:127.0.0.1:41361_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 4256100 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4256100 INFO  (qtp2037187744-36363) [n:127.0.0.1:41041_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVers

[...truncated too long message...]

lrSlf4jReporterTest, CustomHighlightComponentTest, TestUpdate, BlockJoinFacetRandomTest, ClassificationUpdateProcessorIntegrationTest, TlogReplayBufferedWhileIndexingTest, TestDownShardTolerantSearch, InfixSuggestersTest, FieldMutatingUpdateProcessorTest, TestExceedMaxTermLength, TestTrackingShardHandlerFactory, AutoscalingHistoryHandlerTest, PluginInfoTest, TestDynamicLoading, AtomicUpdateProcessorFactoryTest, CdcrBidirectionalTest, CollectionTooManyReplicasTest, CurrencyRangeFacetCloudTest, RecoveryAfterSoftCommitTest, TestChildDocTransformer, OverseerTaskQueueTest, MetricsConfigTest, TestMacros, TestCollectionsAPIViaSolrCloudCluster, TestCloudSearcherWarming, CdcrWithNodesRestartsTest, BooleanFieldTest, BasicFunctionalityTest, TestSimLargeCluster, TestShardHandlerFactory, TestInfoStreamLogging, SolrMetricReporterTest, SaslZkACLProviderTest, BigEndianAscendingWordDeserializerTest, TestSolrCloudWithHadoopAuthPlugin, OutOfBoxZkACLAndCredentialsProvidersTest, LeaderTragicEventTest, TestSQLHandlerNonCloud, ResponseHeaderTest, TestReqParamsAPI, TestManagedSynonymGraphFilterFactory, CoreAdminOperationTest, FastVectorHighlighterTest, AnalysisErrorHandlingTest, SortSpecParsingTest, SearchRateTriggerIntegrationTest, TestMaxTokenLenTokenizer, DocumentAnalysisRequestHandlerTest, ShufflingReplicaListTransformerTest, DistributedQueryComponentOptimizationTest, XmlUpdateRequestHandlerTest, PrimitiveFieldTypeTest, TestFieldCollectionResource, DirectSolrConnectionTest, UUIDFieldTest, SpellCheckCollatorTest, ReplaceNodeNoTargetTest, TestCollationFieldDocValues, CdcrReplicationHandlerTest, DefaultValueUpdateProcessorTest, DistributedExpandComponentTest, TestSystemCollAutoCreate, TestExactSharedStatsCache, TestEmbeddedSolrServerSchemaAPI, SimpleMLTQParserTest, TaggerTest, SyncSliceTest, EmbeddedSolrNoSerializeTest, TestManagedResourceStorage, TestFieldSortValues, SchemaVersionSpecificBehaviorTest, TestRecoveryHdfs, DistribJoinFromCollectionTest, BasicDistributedZkTest, TestCloudPhrasesIdentificationComponent, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestDistributedMap, TestHashPartitioner, TestLockTree, TestMiniSolrCloudClusterSSL, TestPullReplica, TestPullReplicaErrorHandling, TestShortCircuitedRequests, TestSizeLimitedDistributedMap, TestSolrCloudWithKerberosAlt, TestStressInPlaceUpdates, TestStressLiveNodes, NodeAddedTriggerTest, ScheduledTriggerIntegrationTest, ScheduledTriggerTest, TestSimComputePlanAction, TestSimDistribStateManager, TestSimDistributedQueue, TestSimPolicyCloud, HdfsChaosMonkeySafeLeaderTest, HdfsUnloadDistributedZkTest]
   [junit4] Completed [570/835 (2!)] on J1 in 259.86s, 1 test, 1 error <<< FAILURES!

[...truncated 46670 lines...]
-ecj-javadoc-lint-src:
    [mkdir] Created dir: /tmp/ecj478346719
 [ecj-lint] Compiling 1235 source files to /tmp/ecj478346719
 [ecj-lint] Processing annotations
 [ecj-lint] Annotations processed
 [ecj-lint] Processing annotations
 [ecj-lint] No elements to process
 [ecj-lint] invalid Class-Path header in manifest of jar file: /home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: /home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] ----------
 [ecj-lint] 1. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/java/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.java (at line 219)
 [ecj-lint] return (NamedList<Object>) new JavaBinCodec(resolver).unmarshal(in);
 [ecj-lint]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 2. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/java/org/apache/solr/cloud/api/collections/RestoreCmd.java (at line 257)
 [ecj-lint] throw new SolrException(ErrorCode.BAD_REQUEST, "Unexpected number of replicas, replicationFactor, " +
 [ecj-lint]               Replica.Type.NRT + " or " + Replica.Type.TLOG + " must be greater than 0");
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'repository' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 3. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/java/org/apache/solr/handler/loader/JavabinLoader.java (at line 137)
 [ecj-lint] new JavaBinCodec() {
 [ecj-lint]       SolrParams params;
 [ecj-lint]       AddUpdateCommand addCmd = null;
 [ecj-lint]
 [ecj-lint]       @Override
 [ecj-lint]       public List<Object> readIterator(DataInputInputStream fis) throws IOException {
 [ecj-lint]         while (true) {
 [ecj-lint]           Object o = readVal(fis);
 [ecj-lint]           if (o == END_OBJ) break;
 [ecj-lint]           if (o instanceof NamedList) {
 [ecj-lint]             params = ((NamedList) o).toSolrParams();
 [ecj-lint]           } else {
 [ecj-lint]             try {
 [ecj-lint]               if (o instanceof byte[]) {
 [ecj-lint]                 if (params != null) req.setParams(params);
 [ecj-lint]                 byte[] buf = (byte[]) o;
 [ecj-lint]                 contentStreamLoader.load(req, rsp, new ContentStreamBase.ByteArrayStream(buf, null), processor);
 [ecj-lint]               } else {
 [ecj-lint]                 throw new RuntimeException("unsupported type ");
 [ecj-lint]               }
 [ecj-lint]             } catch (Exception e) {
 [ecj-lint]               throw new RuntimeException(e);
 [ecj-lint]             } finally {
 [ecj-lint]               params = null;
 [ecj-lint]               req.setParams(old);
 [ecj-lint]             }
 [ecj-lint]           }
 [ecj-lint]         }
 [ecj-lint]         return Collections.emptyList();
 [ecj-lint]       }
 [ecj-lint]
 [ecj-lint]     }.unmarshal(in);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 4. ERROR in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/java/org/apache/solr/update/DocumentBuilder.java (at line 32)
 [ecj-lint] import org.apache.solr.common.util.ByteArrayUtf8CharSequence;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.common.util.ByteArrayUtf8CharSequence is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 5. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/java/org/apache/solr/update/UpdateLog.java (at line 1867)
 [ecj-lint] if (exceptionOnExecuteUpdate.get() != null) throw exceptionOnExecuteUpdate.get();
 [ecj-lint]                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'proc' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 6. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/java/org/apache/solr/util/FileUtils.java (at line 50)
 [ecj-lint] in = new FileInputStream(src).getChannel();
 [ecj-lint]     ^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] 7. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/java/org/apache/solr/util/FileUtils.java (at line 51)
 [ecj-lint] out = new FileOutputStream(destination).getChannel();
 [ecj-lint]      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] 7 problems (1 error, 6 warnings)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:651: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:101: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build.xml:680: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/common-build.xml:2093: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/common-build.xml:2132: Compile failed; see the compiler error output for details.

Total time: 525 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 428 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/428/

1 tests failed.
FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:41386/forceleader_test_collection

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:41386/forceleader_test_collection
        at __randomizedtesting.SeedInfo.seed([E307B23DADC3C871:59086FD94413110]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
        at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:294)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1063)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1035)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 52 more




Build Log:
[...truncated 13935 lines...]
   [junit4] Suite: org.apache.solr.cloud.ForceLeaderTest
   [junit4]   2> 1600507 INFO  (SUITE-ForceLeaderTest-seed#[E307B23DADC3C871]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/init-core-data-001
   [junit4]   2> 1600508 WARN  (SUITE-ForceLeaderTest-seed#[E307B23DADC3C871]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=33 numCloses=33
   [junit4]   2> 1600508 INFO  (SUITE-ForceLeaderTest-seed#[E307B23DADC3C871]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1600511 INFO  (SUITE-ForceLeaderTest-seed#[E307B23DADC3C871]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 1600511 INFO  (SUITE-ForceLeaderTest-seed#[E307B23DADC3C871]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4] IGNOR/A 0.00s J2 | ForceLeaderTest.testReplicasInLowerTerms
   [junit4]    > Assumption #1: 'badapple' test group is disabled (@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 1600518 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1600519 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1600519 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1600618 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer start zk server on port:33960
   [junit4]   2> 1600618 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:33960
   [junit4]   2> 1600619 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 33960
   [junit4]   2> 1600622 INFO  (zkConnectionManagerCallback-6758-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1600624 INFO  (zkConnectionManagerCallback-6760-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1600626 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1600627 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1600628 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1600630 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1600631 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1600632 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1600633 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1600634 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1600641 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1600642 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1600644 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1600645 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1600814 WARN  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1600814 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1600814 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1600814 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1600815 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1600815 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1600816 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1600816 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@93fd423{/,null,AVAILABLE}
   [junit4]   2> 1600818 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@696bdb8c{HTTP/1.1,[http/1.1]}{127.0.0.1:43336}
   [junit4]   2> 1600818 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.e.j.s.Server Started @1600874ms
   [junit4]   2> 1600818 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=41386, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/control-001/cores}
   [junit4]   2> 1600819 ERROR (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1600819 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1600819 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1600819 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1600819 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1600819 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-10T02:40:05.752Z
   [junit4]   2> 1600821 INFO  (zkConnectionManagerCallback-6762-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1600822 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1600822 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/control-001/solr.xml
   [junit4]   2> 1600827 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1600827 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1600829 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1601010 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33960/solr
   [junit4]   2> 1601012 INFO  (zkConnectionManagerCallback-6766-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1601015 INFO  (zkConnectionManagerCallback-6768-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1601117 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [n:127.0.0.1:41386_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41386_
   [junit4]   2> 1601117 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [n:127.0.0.1:41386_    ] o.a.s.c.Overseer Overseer (id=72596482105737220-127.0.0.1:41386_-n_0000000000) starting
   [junit4]   2> 1601123 INFO  (zkConnectionManagerCallback-6775-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1601126 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [n:127.0.0.1:41386_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33960/solr ready
   [junit4]   2> 1601127 INFO  (OverseerStateUpdate-72596482105737220-127.0.0.1:41386_-n_0000000000) [n:127.0.0.1:41386_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:41386_
   [junit4]   2> 1601127 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [n:127.0.0.1:41386_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41386_
   [junit4]   2> 1601129 INFO  (OverseerStateUpdate-72596482105737220-127.0.0.1:41386_-n_0000000000) [n:127.0.0.1:41386_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1601130 INFO  (zkCallback-6774-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1601161 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [n:127.0.0.1:41386_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1601187 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [n:127.0.0.1:41386_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1601202 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [n:127.0.0.1:41386_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1601203 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [n:127.0.0.1:41386_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1601223 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [n:127.0.0.1:41386_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/control-001/cores
   [junit4]   2> 1601252 INFO  (zkConnectionManagerCallback-6781-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1601253 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1601254 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33960/solr ready
   [junit4]   2> 1601255 INFO  (SocketProxy-Acceptor-41386) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=53204,localport=41386], receiveBufferSize:531000
   [junit4]   2> 1601256 INFO  (SocketProxy-Acceptor-41386) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43336,localport=39868], receiveBufferSize=530904
   [junit4]   2> 1601256 INFO  (qtp1068768018-34041) [n:127.0.0.1:41386_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:41386_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1601260 INFO  (OverseerThreadFactory-7541-thread-1-processing-n:127.0.0.1:41386_) [n:127.0.0.1:41386_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1601369 INFO  (SocketProxy-Acceptor-41386) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=53208,localport=41386], receiveBufferSize:531000
   [junit4]   2> 1601370 INFO  (SocketProxy-Acceptor-41386) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43336,localport=39872], receiveBufferSize=530904
   [junit4]   2> 1601371 INFO  (qtp1068768018-34042) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1601372 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1601373 INFO  (qtp1068768018-34039) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1601374 INFO  (qtp1068768018-34040) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1601374 INFO  (qtp1068768018-34042) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1601378 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1601379 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1602399 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1602414 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1602540 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1602558 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1602559 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1602559 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1602562 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=24, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1602564 WARN  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1602629 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1602629 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1602632 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1602632 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1602634 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@28bb2031
   [junit4]   2> 1602634 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5f5d88ba[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1602635 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1602636 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1602637 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1602637 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1622239354625720320
   [junit4]   2> 1602641 INFO  (searcherExecutor-7546-thread-1-processing-n:127.0.0.1:41386_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@5f5d88ba[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1602644 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 1602644 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 1602648 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1602648 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1602648 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41386/control_collection_shard1_replica_n1/
   [junit4]   2> 1602648 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1602648 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:41386/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1602648 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1602649 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/72596482105737220-core_node2-n_0000000000
   [junit4]   2> 1602651 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41386/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1602753 INFO  (zkCallback-6767-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1602754 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1602756 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1378
   [junit4]   2> 1602760 INFO  (qtp1068768018-34041) [n:127.0.0.1:41386_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 1602857 INFO  (zkCallback-6767-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1602857 INFO  (zkCallback-6767-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1602858 INFO  (qtp1068768018-34041) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:41386_&wt=javabin&version=2} status=0 QTime=1602
   [junit4]   2> 1602867 INFO  (zkConnectionManagerCallback-6786-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1602869 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1602870 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33960/solr ready
   [junit4]   2> 1602870 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1602870 INFO  (SocketProxy-Acceptor-41386) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=53216,localport=41386], receiveBufferSize:531000
   [junit4]   2> 1602871 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1602871 INFO  (SocketProxy-Acceptor-41386) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43336,localport=39880], receiveBufferSize=530904
   [junit4]   2> 1602874 INFO  (OverseerCollectionConfigSetProcessor-72596482105737220-127.0.0.1:41386_-n_0000000000) [n:127.0.0.1:41386_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1602874 INFO  (OverseerThreadFactory-7541-thread-2-processing-n:127.0.0.1:41386_) [n:127.0.0.1:41386_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 1603078 WARN  (OverseerThreadFactory-7541-thread-2-processing-n:127.0.0.1:41386_) [n:127.0.0.1:41386_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 1603081 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 1603089 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=218
   [junit4]   2> 1603091 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=3
   [junit4]   2> 1603272 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-1-001 of type NRT
   [junit4]   2> 1603282 WARN  (closeThreadPool-6787-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1603282 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1603282 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1603282 INFO  (closeThreadPool-6787-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1603337 INFO  (closeThreadPool-6787-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1603337 INFO  (closeThreadPool-6787-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1603337 INFO  (closeThreadPool-6787-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1603339 INFO  (closeThreadPool-6787-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@61d40e0c{/,null,AVAILABLE}
   [junit4]   2> 1603340 INFO  (closeThreadPool-6787-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3379b7a2{HTTP/1.1,[http/1.1]}{127.0.0.1:38825}
   [junit4]   2> 1603340 INFO  (closeThreadPool-6787-thread-1) [    ] o.e.j.s.Server Started @1603396ms
   [junit4]   2> 1603340 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=40178, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-1-001/cores}
   [junit4]   2> 1603341 ERROR (closeThreadPool-6787-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1603341 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1603341 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1603341 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1603341 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1603341 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-10T02:40:08.274Z
   [junit4]   2> 1603361 INFO  (zkConnectionManagerCallback-6789-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1603362 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1603362 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-1-001/solr.xml
   [junit4]   2> 1603367 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1603367 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1603369 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1603539 INFO  (closeThreadPool-6787-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33960/solr
   [junit4]   2> 1603549 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-2-001 of type NRT
   [junit4]   2> 1603549 INFO  (zkConnectionManagerCallback-6793-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1603565 INFO  (zkConnectionManagerCallback-6795-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1603565 WARN  (closeThreadPool-6787-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1603565 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1603566 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1603566 INFO  (closeThreadPool-6787-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1603569 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1603573 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.c.ZkController Publish node=127.0.0.1:40178_ as DOWN
   [junit4]   2> 1603574 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1603574 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40178_
   [junit4]   2> 1603576 INFO  (zkCallback-6774-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1603576 INFO  (zkCallback-6767-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1603617 INFO  (closeThreadPool-6787-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1603586 INFO  (zkCallback-6785-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1603617 INFO  (closeThreadPool-6787-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1603618 INFO  (closeThreadPool-6787-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1603619 INFO  (closeThreadPool-6787-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5702de8e{/,null,AVAILABLE}
   [junit4]   2> 1603620 INFO  (closeThreadPool-6787-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4bf19fdb{HTTP/1.1,[http/1.1]}{127.0.0.1:36570}
   [junit4]   2> 1603620 INFO  (closeThreadPool-6787-thread-2) [    ] o.e.j.s.Server Started @1603676ms
   [junit4]   2> 1603620 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=38384, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-2-001/cores}
   [junit4]   2> 1603620 ERROR (closeThreadPool-6787-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1603621 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1603621 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1603621 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1603621 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1603621 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-10T02:40:08.554Z
   [junit4]   2> 1603621 INFO  (zkCallback-6794-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1603649 INFO  (zkConnectionManagerCallback-6799-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1603650 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1603650 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-2-001/solr.xml
   [junit4]   2> 1603655 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1603655 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1603657 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1603705 INFO  (zkConnectionManagerCallback-6804-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1603706 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1603707 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33960/solr ready
   [junit4]   2> 1603719 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1603763 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1603767 INFO  (closeThreadPool-6787-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33960/solr
   [junit4]   2> 1603773 INFO  (zkConnectionManagerCallback-6809-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1603787 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1603787 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1603789 INFO  (closeThreadPool-6787-thread-1) [n:127.0.0.1:40178_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-1-001/cores
   [junit4]   2> 1603790 INFO  (zkConnectionManagerCallback-6811-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1603797 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1603800 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.c.ZkController Publish node=127.0.0.1:38384_ as DOWN
   [junit4]   2> 1603801 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1603801 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38384_
   [junit4]   2> 1603802 INFO  (zkCallback-6785-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1603803 INFO  (zkCallback-6774-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1603803 INFO  (zkCallback-6767-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1603807 INFO  (zkCallback-6794-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1603817 INFO  (zkCallback-6810-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1603825 INFO  (zkCallback-6803-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1603853 INFO  (zkConnectionManagerCallback-6818-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1603854 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1603856 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33960/solr ready
   [junit4]   2> 1603860 INFO  (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[E307B23DADC3C871]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-3-001 of type NRT
   [junit4]   2> 1603861 WARN  (closeThreadPool-6787-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1603862 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1603862 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1603862 INFO  (closeThreadPool-6787-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1603862 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1603867 INFO  (closeThreadPool-6787-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1603867 INFO  (closeThreadPool-6787-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1603867 INFO  (closeThreadPool-6787-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1603867 INFO  (closeThreadPool-6787-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@44c0c869{/,null,AVAILABLE}
   [junit4]   2> 1603868 INFO  (closeThreadPool-6787-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3e7ad02c{HTTP/1.1,[http/1.1]}{127.0.0.1:35378}
   [junit4]   2> 1603868 INFO  (closeThreadPool-6787-thread-3) [    ] o.e.j.s.Server Started @1603923ms
   [junit4]   2> 1603868 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=34394, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-3-001/cores}
   [junit4]   2> 1603868 ERROR (closeThreadPool-6787-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1603868 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1603868 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 1603868 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1603868 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1603868 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-01-10T02:40:08.801Z
   [junit4]   2> 1603874 INFO  (zkConnectionManagerCallback-6821-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1603875 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1603875 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-3-001/solr.xml
   [junit4]   2> 1603879 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1603880 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1603889 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1603894 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1603911 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1603911 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1603913 INFO  (closeThreadPool-6787-thread-2) [n:127.0.0.1:38384_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-2-001/cores
   [junit4]   2> 1604294 INFO  (closeThreadPool-6787-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33960/solr
   [junit4]   2> 1604296 INFO  (zkConnectionManagerCallback-6825-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1604301 INFO  (zkConnectionManagerCallback-6827-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1604306 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1604310 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.c.ZkController Publish node=127.0.0.1:34394_ as DOWN
   [junit4]   2> 1604311 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1604311 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34394_
   [junit4]   2> 1604312 INFO  (zkCallback-6794-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1604312 INFO  (zkCallback-6774-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1604312 INFO  (zkCallback-6785-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1604313 INFO  (zkCallback-6767-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1604322 INFO  (zkCallback-6803-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1604322 INFO  (zkCallback-6810-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1604322 INFO  (zkCallback-6826-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1604323 INFO  (zkCallback-6817-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1604340 INFO  (zkConnectionManagerCallback-6834-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1604341 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1604342 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33960/solr ready
   [junit4]   2> 1604343 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1604389 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1604404 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1604404 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1604406 INFO  (closeThreadPool-6787-thread-3) [n:127.0.0.1:34394_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-3-001/cores
   [junit4]   2> 1604468 INFO  (SocketProxy-Acceptor-34394) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=35406,localport=34394], receiveBufferSize:531000
   [junit4]   2> 1604469 INFO  (SocketProxy-Acceptor-34394) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=35378,localport=46784], receiveBufferSize=530904
   [junit4]   2> 1604469 INFO  (SocketProxy-Acceptor-34394) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=35404,localport=34394], receiveBufferSize:531000
   [junit4]   2> 1604469 INFO  (qtp1338011078-34170) [n:127.0.0.1:34394_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:38384_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1604470 INFO  (SocketProxy-Acceptor-34394) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=35378,localport=46788], receiveBufferSize=530904
   [junit4]   2> 1604470 INFO  (SocketProxy-Acceptor-34394) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=35410,localport=34394], receiveBufferSize:531000
   [junit4]   2> 1604471 INFO  (qtp1338011078-34173) [n:127.0.0.1:34394_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34394_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1604471 INFO  (SocketProxy-Acceptor-34394) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=35378,localport=46790], receiveBufferSize=530904
   [junit4]   2> 1604472 INFO  (qtp1338011078-34172) [n:127.0.0.1:34394_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:40178_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1604473 INFO  (OverseerCollectionConfigSetProcessor-72596482105737220-127.0.0.1:41386_-n_0000000000) [n:127.0.0.1:41386_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1604476 INFO  (SocketProxy-Acceptor-34394) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=35416,localport=34394], receiveBufferSize:531000
   [junit4]   2> 1604477 INFO  (SocketProxy-Acceptor-34394) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=35378,localport=46794], receiveBufferSize=530904
   [junit4]   2> 1604477 INFO  (qtp1338011078-34175) [n:127.0.0.1:34394_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604479 INFO  (qtp1338011078-34174) [n:127.0.0.1:34394_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604480 INFO  (qtp1338011078-34175) [n:127.0.0.1:34394_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604481 INFO  (qtp1338011078-34174) [n:127.0.0.1:34394_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604482 INFO  (qtp1338011078-34175) [n:127.0.0.1:34394_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604482 INFO  (SocketProxy-Acceptor-38384) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=49608,localport=38384], receiveBufferSize:531000
   [junit4]   2> 1604483 INFO  (SocketProxy-Acceptor-38384) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36570,localport=53820], receiveBufferSize=530904
   [junit4]   2> 1604484 INFO  (qtp1161599465-34131) [n:127.0.0.1:38384_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604485 INFO  (qtp1161599465-34132) [n:127.0.0.1:38384_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604485 INFO  (qtp1161599465-34133) [n:127.0.0.1:38384_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604486 INFO  (qtp1161599465-34134) [n:127.0.0.1:38384_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604488 INFO  (qtp1161599465-34130) [n:127.0.0.1:38384_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604488 INFO  (SocketProxy-Acceptor-40178) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=36404,localport=40178], receiveBufferSize:531000
   [junit4]   2> 1604488 INFO  (SocketProxy-Acceptor-40178) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=38825,localport=46184], receiveBufferSize=530904
   [junit4]   2> 1604490 INFO  (qtp1748820213-34108) [n:127.0.0.1:40178_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604490 INFO  (qtp1748820213-34109) [n:127.0.0.1:40178_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604491 INFO  (qtp1748820213-34110) [n:127.0.0.1:40178_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604492 INFO  (qtp1748820213-34111) [n:127.0.0.1:40178_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604493 INFO  (qtp1748820213-34107) [n:127.0.0.1:40178_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604494 INFO  (qtp1068768018-34041) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1604495 INFO  (qtp1068768018-34039) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1604495 INFO  (qtp1068768018-34040) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1604496 INFO  (qtp1068768018-34042) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604498 INFO  (qtp1068768018-34043) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604499 INFO  (qtp1068768018-34041) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604500 INFO  (qtp1068768018-34039) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604501 INFO  (qtp1068768018-34040) [n:127.0.0.1:41386_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1604502 INFO  (OverseerThreadFactory-7541-thread-3-processing-n:127.0.0.1:41386_) [n:127.0.0.1:41386_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:38384_ for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 1604502 INFO  (OverseerThreadFactory-7541-thread-4-processing-n:127.0.0.1:41386_) [n:127.0.0.1:41386_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:34394_ for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 1604504 INFO  (OverseerThreadFactory-7541-thread-3-processing-n:127.0.0.1:41386_) [n:127.0.0.1:41386_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 1604505 INFO  (SocketProxy-Acceptor-38384) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=49616,localport=38384], receiveBufferSize:531000
   [junit4]   2> 1604506 INFO  (OverseerThreadFactory-7541-thread-4-processing-n:127.0.0.1:41386_) [n:127.0.0.1:41386_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
   [junit4]   2> 1604506 INFO  (SocketProxy-Acceptor-38384) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36570,localport=53828], receiveBufferSize=530904
   [junit4]   2> 1604506 INFO  (qtp1161599465-34132) [n:127.0.0.1:38384_    x:collection1_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n1&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1604509 INFO  (SocketProxy-Acceptor-34394) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=35432,localport=34394], receiveBufferSize:531000
   [junit4]   2> 1604509 INFO  (SocketProxy-Acceptor-34394) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=35378,localport=46810], receiveBufferSize=530904
   [junit4]   2> 1604510 INFO  (qtp1338011078-34175) [n:127.0.0.1:34394_    x:collection1_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n2&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1605527 INFO  (qtp1338011078-34175) [n:127.0.0.1:34394_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1605530 INFO  (qtp1161599465-34132) [n:127.0.0.1:38384_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 1605543 INFO  (qtp1338011078-34175) [n:127.0.0.1:34394_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.s.IndexSchema [collection1_shard2_replica_n2] Schema name=test
   [junit4]   2> 1605580 INFO  (qtp1161599465-34132) [n:127.0.0.1:38384_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=test
   [junit4]   2> 1605679 INFO  (qtp1338011078-34175) [n:127.0.0.1:34394_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1605704 INFO  (qtp1161599465-34132) [n:127.0.0.1:38384_ c:collection1 s:shard1  x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1605720 INFO  (qtp1338011078-34175) [n:127.0.0.1:34394_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n2' using configuration from collection collection1, trusted=true
   [junit4]   2> 1605721 INFO  (qtp1338011078-34175) [n:127.0.0.1:34394_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n2' (registry 'solr.core.collection1.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b04bb1
   [junit4]   2> 1605721 INFO  (qtp1338011078-34175) [n:127.0.0.1:34394_ c:collection1 s:shard2  x:collection1_shard2_replica_n2] o.a.s.c.SolrCore [[collection1_shard2_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ForceLeaderTest_E307B23DADC3C871-001/shard-3-001/cores/collection1_shard2_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/ch