[JENKINS] Lucene-Solr-Tests-7.x - Build # 155 - Unstable

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 155 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/155/

6 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Error from server at http://127.0.0.1:45081/b_/yh/collection1: java.lang.NullPointerException  at org.apache.solr.search.grouping.distributed.responseprocessor.StoredFieldsShardResponseProcessor.process(StoredFieldsShardResponseProcessor.java:41)  at org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:768)  at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:751)  at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:423)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2483)  at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:45081/b_/yh/collection1: java.lang.NullPointerException
        at org.apache.solr.search.grouping.distributed.responseprocessor.StoredFieldsShardResponseProcessor.process(StoredFieldsShardResponseProcessor.java:41)
        at org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:768)
        at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:751)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:423)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2483)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

        at __randomizedtesting.SeedInfo.seed([24D52AB15B73465F:AC81156BF58F2BA7]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at org.apache.solr.TestDistributedSearch.queryRandomUpServer(TestDistributedSearch.java:1168)
        at org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:1130)
        at org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:992)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1019)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CloudExitableDirectoryReaderTest

Error Message:
Error from server at https://127.0.0.1:39244/solr: KeeperErrorCode = Session expired for /overseer/collection-map-completed/mn-1273090e-6376-471c-b093-3474c2f11749

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:39244/solr: KeeperErrorCode = Session expired for /overseer/collection-map-completed/mn-1273090e-6376-471c-b093-3474c2f11749
        at __randomizedtesting.SeedInfo.seed([24D52AB15B73465F]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.client.solrj.request.CollectionAdminRequest$RequestStatus.waitFor(CollectionAdminRequest.java:1203)
        at org.apache.solr.client.solrj.request.CollectionAdminRequest$AsyncCollectionAdminRequest.processAndWait(CollectionAdminRequest.java:172)
        at org.apache.solr.client.solrj.request.CollectionAdminRequest$AsyncCollectionAdminRequest.processAndWait(CollectionAdminRequest.java:158)
        at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.setupCluster(CloudExitableDirectoryReaderTest.java:48)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.LeaderElectionContextKeyTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([24D52AB15B73465F:AC81156BF58F2BA7]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitStaticIndexReplication

Error Message:
Shard shard1_0 replicas do not have same number of documents expected:<11> but was:<0>

Stack Trace:
java.lang.AssertionError: Shard shard1_0 replicas do not have same number of documents expected:<11> but was:<0>
        at __randomizedtesting.SeedInfo.seed([24D52AB15B73465F:6E9FBEB1CADB695A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.ShardSplitTest.assertConsistentReplicas(ShardSplitTest.java:243)
        at org.apache.solr.cloud.ShardSplitTest.testSplitStaticIndexReplication(ShardSplitTest.java:217)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([24D52AB15B73465F:38D4573C2ED638CC]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertFalse(Assert.java:68)
        at org.junit.Assert.assertFalse(Assert.java:79)
        at org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes(TestTlogReplica.java:497)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.schema.TestPointFields.testDoublePointFieldSortAndFunction

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([24D52AB15B73465F:667CBD1283FB20B1]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:884)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:844)
        at org.apache.solr.schema.TestPointFields.doTestDoublePointFunctionQuery(TestPointFields.java:3354)
        at org.apache.solr.schema.TestPointFields.testDoublePointFieldSortAndFunction(TestPointFields.java:561)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//*[@numFound='20']
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="10" start="0"><doc><str name="id">A</str><double name="number_p_d">-10.21574592590332</double><double name="field(number_p_d)">-10.21574592590332</double></doc><doc><str name="id">B</str><double name="number_p_d">-4.977153107652802E-31</double><double name="field(number_p_d)">-4.977153107652802E-31</double></doc><doc><str name="id">C</str><double name="number_p_d">-5.584665245805809E-7</double><double name="field(number_p_d)">-5.584665245805809E-7</double></doc><doc><str name="id">D</str><double name="number_p_d">-5.900989916608113E-30</double><double name="field(number_p_d)">-5.900989916608113E-30</double></doc><doc><str name="id">E</str><double name="number_p_d">-1.1146119682869515E-37</double><double name="field(number_p_d)">-1.1146119682869515E-37</double></doc><doc><str name="id">F</str><double name="number_p_d">3.4120297754651516E-30</double><double name="field(number_p_d)">3.4120297754651516E-30</double></doc><doc><str name="id">G</str><double name="number_p_d">9.272587595865947E28</double><double name="field(number_p_d)">9.272587595865947E28</double></doc><doc><str name="id">H</str><double name="number_p_d">-8.420276683527713E-31</double><double name="field(number_p_d)">-8.420276683527713E-31</double></doc><doc><str name="id">I</str><double name="number_p_d">-1.2369949899948551E-6</double><double name="field(number_p_d)">-1.2369949899948551E-6</double></doc><doc><str name="id">J</str><double name="number_p_d">-107473.546875</double><double name="field(number_p_d)">-107473.546875</double></doc></result>
</response>

        request was:q=*:*&fl=id,+number_p_d,+field(number_p_d)&sort=id+asc&rows=20&wt=xml
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:877)
        ... 42 more




Build Log:
[...truncated 11318 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4]   2> 63224 INFO  (SUITE-TestTlogReplica-seed#[24D52AB15B73465F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_24D52AB15B73465F-001/init-core-data-001
   [junit4]   2> 63269 WARN  (SUITE-TestTlogReplica-seed#[24D52AB15B73465F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 63270 INFO  (SUITE-TestTlogReplica-seed#[24D52AB15B73465F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 63288 INFO  (SUITE-TestTlogReplica-seed#[24D52AB15B73465F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 64736 INFO  (SUITE-TestTlogReplica-seed#[24D52AB15B73465F]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_24D52AB15B73465F-001/tempDir-001
   [junit4]   2> 64736 INFO  (SUITE-TestTlogReplica-seed#[24D52AB15B73465F]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 64757 INFO  (Thread-30) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 64758 INFO  (Thread-30) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 64846 INFO  (SUITE-TestTlogReplica-seed#[24D52AB15B73465F]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:41573
   [junit4]   2> 64811 ERROR (Thread-30) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 65459 INFO  (jetty-launcher-27-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 65459 INFO  (jetty-launcher-27-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 65534 INFO  (jetty-launcher-27-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@14d4c364{/solr,null,AVAILABLE}
   [junit4]   2> 65680 INFO  (jetty-launcher-27-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@793cc695{/solr,null,AVAILABLE}
   [junit4]   2> 65755 INFO  (jetty-launcher-27-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@579d975a{SSL,[ssl, http/1.1]}{127.0.0.1:40128}
   [junit4]   2> 65755 INFO  (jetty-launcher-27-thread-2) [    ] o.e.j.s.Server Started @76655ms
   [junit4]   2> 65755 INFO  (jetty-launcher-27-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40128}
   [junit4]   2> 65755 ERROR (jetty-launcher-27-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 65755 INFO  (jetty-launcher-27-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 65755 INFO  (jetty-launcher-27-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 65755 INFO  (jetty-launcher-27-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 65756 INFO  (jetty-launcher-27-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-11T16:32:06.575Z
   [junit4]   2> 65864 INFO  (jetty-launcher-27-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@11618996{SSL,[ssl, http/1.1]}{127.0.0.1:34100}
   [junit4]   2> 65865 INFO  (jetty-launcher-27-thread-1) [    ] o.e.j.s.Server Started @76765ms
   [junit4]   2> 65865 INFO  (jetty-launcher-27-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34100}
   [junit4]   2> 65865 ERROR (jetty-launcher-27-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 65865 INFO  (jetty-launcher-27-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 65865 INFO  (jetty-launcher-27-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 65865 INFO  (jetty-launcher-27-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 65865 INFO  (jetty-launcher-27-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-11T16:32:06.685Z
   [junit4]   2> 65934 INFO  (jetty-launcher-27-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 65948 INFO  (jetty-launcher-27-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 66236 INFO  (jetty-launcher-27-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41573/solr
   [junit4]   2> 66254 INFO  (jetty-launcher-27-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41573/solr
   [junit4]   2> 66555 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e71c899da0003, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 67035 INFO  (jetty-launcher-27-thread-2) [n:127.0.0.1:40128_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 67036 INFO  (jetty-launcher-27-thread-2) [n:127.0.0.1:40128_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40128_solr
   [junit4]   2> 67082 INFO  (jetty-launcher-27-thread-2) [n:127.0.0.1:40128_solr    ] o.a.s.c.Overseer Overseer (id=98641348237328389-127.0.0.1:40128_solr-n_0000000000) starting
   [junit4]   2> 67187 INFO  (jetty-launcher-27-thread-2) [n:127.0.0.1:40128_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40128_solr
   [junit4]   2> 67341 INFO  (OverseerStateUpdate-98641348237328389-127.0.0.1:40128_solr-n_0000000000) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 67690 INFO  (jetty-launcher-27-thread-1) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 67737 INFO  (jetty-launcher-27-thread-1) [n:127.0.0.1:34100_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 67739 INFO  (jetty-launcher-27-thread-1) [n:127.0.0.1:34100_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34100_solr
   [junit4]   2> 67784 INFO  (zkCallback-38-thread-1-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 67801 INFO  (zkCallback-39-thread-1-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 67947 INFO  (jetty-launcher-27-thread-2) [n:127.0.0.1:40128_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40128.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@18892b93
   [junit4]   2> 67983 INFO  (jetty-launcher-27-thread-2) [n:127.0.0.1:40128_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40128.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@18892b93
   [junit4]   2> 67983 INFO  (jetty-launcher-27-thread-2) [n:127.0.0.1:40128_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40128.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@18892b93
   [junit4]   2> 67997 INFO  (jetty-launcher-27-thread-2) [n:127.0.0.1:40128_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_24D52AB15B73465F-001/tempDir-001/node2/.
   [junit4]   2> 68740 INFO  (jetty-launcher-27-thread-1) [n:127.0.0.1:34100_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34100.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@18892b93
   [junit4]   2> 68914 INFO  (jetty-launcher-27-thread-1) [n:127.0.0.1:34100_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34100.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@18892b93
   [junit4]   2> 68915 INFO  (jetty-launcher-27-thread-1) [n:127.0.0.1:34100_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34100.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@18892b93
   [junit4]   2> 68916 INFO  (jetty-launcher-27-thread-1) [n:127.0.0.1:34100_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_24D52AB15B73465F-001/tempDir-001/node1/.
   [junit4]   2> 69469 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e71c899da0007, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 69615 INFO  (SUITE-TestTlogReplica-seed#[24D52AB15B73465F]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 69676 INFO  (SUITE-TestTlogReplica-seed#[24D52AB15B73465F]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41573/solr ready
   [junit4]   2> 69680 INFO  (SUITE-TestTlogReplica-seed#[24D52AB15B73465F]-worker) [    ] o.a.s.c.TestTlogReplica Using legacyCloud?: false
   [junit4]   2> 71110 INFO  (qtp1662402658-148) [n:127.0.0.1:34100_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 71175 INFO  (qtp1662402658-148) [n:127.0.0.1:34100_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=64
   [junit4]   2> 71226 INFO  (TEST-TestTlogReplica.testRealTimeGet-seed#[24D52AB15B73465F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRealTimeGet
   [junit4]   2> 71247 INFO  (qtp1662402658-151) [n:127.0.0.1:34100_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_real_time_get&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 71321 INFO  (OverseerThreadFactory-45-thread-1-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.CreateCollectionCmd Create collection tlog_replica_test_real_time_get
   [junit4]   2> 71581 INFO  (OverseerStateUpdate-98641348237328389-127.0.0.1:40128_solr-n_0000000000) [n:127.0.0.1:40128_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_real_time_get",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_real_time_get_shard1_replica_t1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:34100/solr",
   [junit4]   2>   "type":"TLOG"}
   [junit4]   2> 71646 INFO  (OverseerStateUpdate-98641348237328389-127.0.0.1:40128_solr-n_0000000000) [n:127.0.0.1:40128_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_real_time_get",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"tlog_replica_test_real_time_get_shard1_replica_t2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:40128/solr",
   [junit4]   2>   "type":"TLOG"}
   [junit4]   2> 71998 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 71998 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 72261 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 72262 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 72418 INFO  (zkCallback-38-thread-2-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 72418 INFO  (zkCallback-38-thread-1-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 72421 INFO  (zkCallback-39-thread-1-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 72450 INFO  (zkCallback-39-thread-2-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 73153 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 73567 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.s.IndexSchema [tlog_replica_test_real_time_get_shard1_replica_t1] Schema name=minimal
   [junit4]   2> 73640 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 73699 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 73699 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_real_time_get_shard1_replica_t1' using configuration from collection tlog_replica_test_real_time_get, trusted=true
   [junit4]   2> 73700 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34100.solr.core.tlog_replica_test_real_time_get.shard1.replica_t1' (registry 'solr.core.tlog_replica_test_real_time_get.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@18892b93
   [junit4]   2> 73700 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 73700 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SolrCore [[tlog_replica_test_real_time_get_shard1_replica_t1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_24D52AB15B73465F-001/tempDir-001/node1/tlog_replica_test_real_time_get_shard1_replica_t1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_24D52AB15B73465F-001/tempDir-001/node1/./tlog_replica_test_real_time_get_shard1_replica_t1/data/]
   [junit4]   2> 73764 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.s.IndexSchema [tlog_replica_test_real_time_get_shard1_replica_t2] Schema name=minimal
   [junit4]   2> 73812 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 73812 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_real_time_get_shard1_replica_t2' using configuration from collection tlog_replica_test_real_time_get, trusted=true
   [junit4]   2> 73812 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40128.solr.core.tlog_replica_test_real_time_get.shard1.replica_t2' (registry 'solr.core.tlog_replica_test_real_time_get.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@18892b93
   [junit4]   2> 73813 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 73813 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.SolrCore [[tlog_replica_test_real_time_get_shard1_replica_t2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_24D52AB15B73465F-001/tempDir-001/node2/tlog_replica_test_real_time_get_shard1_replica_t2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_24D52AB15B73465F-001/tempDir-001/node2/./tlog_replica_test_real_time_get_shard1_replica_t2/data/]
   [junit4]   2> 74517 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 74517 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 74532 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 74532 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 74546 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 74546 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 74547 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 74548 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 74562 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4e678f22[tlog_replica_test_real_time_get_shard1_replica_t1] main]
   [junit4]   2> 74565 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 74566 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 74597 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 74615 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578261582116290560
   [junit4]   2> 74690 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 74704 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@63064367[tlog_replica_test_real_time_get_shard1_replica_t2] main]
   [junit4]   2> 74709 INFO  (searcherExecutor-50-thread-1-processing-n:127.0.0.1:34100_solr x:tlog_replica_test_real_time_get_shard1_replica_t1 s:shard1 c:tlog_replica_test_real_time_get r:core_node3) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_t1] Registered new searcher Searcher@4e678f22[tlog_replica_test_real_time_get_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 74710 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 74725 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 74726 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 74728 INFO  (zkCallback-38-thread-1-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 74728 INFO  (zkCallback-38-thread-2-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 74831 INFO  (searcherExecutor-51-thread-1-processing-n:127.0.0.1:40128_solr x:tlog_replica_test_real_time_get_shard1_replica_t2 s:shard1 c:tlog_replica_test_real_time_get r:core_node4) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_t2] Registered new searcher Searcher@63064367[tlog_replica_test_real_time_get_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 74838 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578261582350123008
   [junit4]   2> 74899 INFO  (zkCallback-39-thread-2-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 74899 INFO  (zkCallback-39-thread-1-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 75196 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 75196 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 75196 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 75263 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_real_time_get_shard1_replica_t1 url=https://127.0.0.1:34100/solr START replicas=[https://127.0.0.1:40128/solr/tlog_replica_test_real_time_get_shard1_replica_t2/] nUpdates=100
   [junit4]   2> 75570 INFO  (qtp2110719818-157) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=13
   [junit4]   2> 75586 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_real_time_get_shard1_replica_t1 url=https://127.0.0.1:34100/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 75586 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 75586 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 75587 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 75590 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ZkController tlog_replica_test_real_time_get_shard1_replica_t1 stopping background replication from leader
   [junit4]   2> 75609 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/ shard1
   [junit4]   2> 75710 INFO  (zkCallback-38-thread-2-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 75710 INFO  (zkCallback-38-thread-1-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 75720 INFO  (zkCallback-39-thread-2-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 75744 INFO  (zkCallback-39-thread-1-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 75761 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 75822 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=3824
   [junit4]   2> 75856 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.ZkController tlog_replica_test_real_time_get_shard1_replica_t2 starting background replication from leader
   [junit4]   2> 75856 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
   [junit4]   2> 76037 INFO  (zkCallback-38-thread-2-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 76038 INFO  (zkCallback-39-thread-1-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 76038 INFO  (zkCallback-39-thread-2-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 76038 INFO  (zkCallback-38-thread-1-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 76094 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
   [junit4]   2> 76095 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=3834
   [junit4]   2> 76102 INFO  (indexFetcher-61-thread-1) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.IndexFetcher Updated masterUrl to https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 76214 INFO  (zkCallback-38-thread-3-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 76218 INFO  (zkCallback-38-thread-1-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 76218 INFO  (zkCallback-39-thread-2-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 76219 INFO  (zkCallback-39-thread-1-processing-n:127.0.0.1:34100_solr) [n:127.0.0.1:34100_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 76473 INFO  (qtp1662402658-153) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=113
   [junit4]   2> 76474 INFO  (indexFetcher-61-thread-1) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 76474 INFO  (indexFetcher-61-thread-1) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 76474 INFO  (indexFetcher-61-thread-1) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 76474 INFO  (indexFetcher-61-thread-1) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 76501 INFO  (qtp1662402658-151) [n:127.0.0.1:34100_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 76501 INFO  (qtp1662402658-151) [n:127.0.0.1:34100_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_real_time_get&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2} status=0 QTime=5254
   [junit4]   2> 77557 INFO  (OverseerCollectionConfigSetProcessor-98641348237328389-127.0.0.1:40128_solr-n_0000000000) [n:127.0.0.1:40128_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 77596 INFO  (qtp2110719818-161) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2}{add=[0 (1578261584680058880)]} 0 14
   [junit4]   2> 77612 INFO  (qtp1662402658-152) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0 (1578261584680058880)]} 0 949
   [junit4]   2> 77710 INFO  (qtp2110719818-160) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 77805 INFO  (qtp1662402658-149) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&_stateVer_=tlog_replica_test_real_time_get:6&omitHeader=true&shards.purpose=1&NOW=1505147538528&ids=0&isShard=true&shard.url=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=61
   [junit4]   2> 77821 INFO  (qtp2110719818-160) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/get params={qt=/get&_stateVer_=tlog_replica_test_real_time_get:6&ids=0&wt=javabin&version=2} status=0 QTime=113
   [junit4]   2> 77823 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 77882 INFO  (qtp1662402658-155) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505147538643&ids=0&isShard=true&shard.url=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 77883 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=60
   [junit4]   2> 77885 INFO  (qtp2110719818-157) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 77910 INFO  (qtp1662402658-153) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505147538704&ids=0&isShard=true&shard.url=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 77911 INFO  (qtp2110719818-157) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=26
   [junit4]   2> 78054 INFO  (qtp2110719818-161) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2}{add=[1 (1578261585687740416)]} 0 30
   [junit4]   2> 78054 INFO  (qtp1662402658-153) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:40128/solr/tlog_replica_test_real_time_get_shard1_replica_t2/&wt=javabin&version=2}{add=[1 (1578261585687740416)]} 0 47
   [junit4]   2> 78055 INFO  (qtp2110719818-163) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 123
   [junit4]   2> 78081 INFO  (qtp2110719818-161) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 78098 INFO  (qtp1662402658-152) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&_stateVer_=tlog_replica_test_real_time_get:6&omitHeader=true&shards.purpose=1&NOW=1505147538901&ids=1&isShard=true&shard.url=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 78100 INFO  (qtp2110719818-161) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/get params={qt=/get&_stateVer_=tlog_replica_test_real_time_get:6&ids=1&wt=javabin&version=2} status=0 QTime=18
   [junit4]   2> 78126 INFO  (qtp1662402658-148) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 78127 INFO  (qtp1662402658-152) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505147538946&ids=1&isShard=true&shard.url=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 78128 INFO  (qtp1662402658-148) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 78179 INFO  (qtp2110719818-162) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 78181 INFO  (qtp1662402658-155) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505147538999&ids=1&isShard=true&shard.url=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 78195 INFO  (qtp2110719818-162) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=16
   [junit4]   2> 78197 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 78198 INFO  (qtp1662402658-151) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505147539016&ids=0,1&isShard=true&shard.url=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
   [junit4]   2> 78199 INFO  (qtp1662402658-154) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={qt=/get&ids=0&ids=1&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 78201 INFO  (qtp2110719818-160) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/
   [junit4]   2> 78239 INFO  (qtp1662402658-151) [n:127.0.0.1:34100_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t1]  webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505147539021&ids=0,1&isShard=true&shard.url=https://127.0.0.1:34100/solr/tlog_replica_test_real_time_get_shard1_replica_t1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=37
   [junit4]   2> 78240 INFO  (qtp2110719818-160) [n:127.0.0.1:40128_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node4 x:tlog_replica_test_real_time_get_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t2]  webapp=/solr path=/get params={qt=/get&ids=0&ids=1&wt=javabin&version=2} status=0 QTime=38
   [junit4]   2> 78241 INFO  (TEST-TestTlogReplica.testRealTimeGet-seed#[24D52AB15B73465F]) [    ] o.a.s.c.TestTlogReplica tearDown deleting collection
   [junit4]   2> 78242 INFO  (qtp1662402658-153) [n:127.0.0.1:34100_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=tlog_replica_test_real_time_get&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 78410 INFO  (OverseerThreadFactory-45-thread-2-processing-n:127.0.0.1:40128_solr) [n:127.0.0.1:40128_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 78454 INFO  (qtp1662402658-152) [n:127.0.0.1:34100_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_t1, tag=null
   [junit4]   2> 78455 INFO  (qtp1662402658-152) [n:127.0.0.1:34100_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2cd3ac6: rootName = solr_34100, domain = solr.core.tlog_replica_test_real_time_get.shard1.replica_t1, service url = null, agent id = null] for registry solr.core.tlog_replica_test_real_time_get.shard1.replica_t1 / com.codahale.metrics.MetricRegistry@135cee64
   [junit4]   2> 78662 INFO  (qtp1662402658-152) [n:127.0.0.1:34100_solr    ] o.a.s.c.ZkController tlog_replica_test_real_time_get_shard1_replica_t1 stopping background replication from leader
   [junit4]   2> 78685 INFO  (qtp2110719818-164) [n:127.0.0.1:40128_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_t2, tag=null
   [junit4]   2> 78685 INFO  (qtp2110719818-164) [n:127.0.0.1:40128_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2a2d0b2c: rootName = solr_40128, domain = solr.core.tlog_replica_test_real_time_get.shard1.replica_t2, service url = null, agent id = null] for registry solr.core.tlog_replica_test_real_time_get.shard1

[...truncated too long message...]

istry=solr.core.collection1, tag=609456709
   [junit4]   2> 3228143 INFO  (coreCloseExecutor-4662-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7afd909c: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@3fd909c
   [junit4]   2> 3228192 INFO  (TEST-TestDistributedSearch.test-seed#[24D52AB15B73465F]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@76e70162{/b_/yh,null,UNAVAILABLE}
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDistributedSearch -Dtests.method=test -Dtests.seed=24D52AB15B73465F -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ar-DZ -Dtests.timezone=EST -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    137s J2 | TestDistributedSearch.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:45081/b_/yh/collection1: java.lang.NullPointerException
   [junit4]    > at org.apache.solr.search.grouping.distributed.responseprocessor.StoredFieldsShardResponseProcessor.process(StoredFieldsShardResponseProcessor.java:41)
   [junit4]    > at org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:768)
   [junit4]    > at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:751)
   [junit4]    > at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:423)
   [junit4]    > at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]    > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2483)
   [junit4]    > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
   [junit4]    > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
   [junit4]    > at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
   [junit4]    > at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]    > at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]    > at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]    > at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]    > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]    > at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]    > at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]    > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]    > at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]    > at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]    > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    > at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]    > at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    > at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]    > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]    > at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]    > at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]    > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]    > at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]    > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]    > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([24D52AB15B73465F:AC81156BF58F2BA7]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
   [junit4]    > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
   [junit4]    > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
   [junit4]    > at org.apache.solr.TestDistributedSearch.queryRandomUpServer(TestDistributedSearch.java:1168)
   [junit4]    > at org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:1130)
   [junit4]    > at org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:992)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1019)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.TestDistributedSearch_24D52AB15B73465F-001
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=16767, maxDocsPerChunk=4, blockSize=681), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=16767, blockSize=681)), sim=RandomSimilarity(queryNorm=true): {}, locale=ar-DZ, timezone=EST
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=154902704,total=531628032
   [junit4]   2> NOTE: All tests run in this JVM: [TestLegacyTerms, PreAnalyzedFieldManagedSchemaCloudTest, TestHdfsUpdateLog, BlockCacheTest, MetricsHandlerTest, SolrCoreCheckLockOnStartupTest, SolrGangliaReporterTest, SharedFSAutoReplicaFailoverTest, RankQueryTest, TestStressReorder, CreateCollectionCleanupTest, TestIndexSearcher, UniqFieldsUpdateProcessorFactoryTest, TestUniqueKeyFieldResource, TestCopyFieldCollectionResource, TestPerFieldSimilarityWithDefaultOverride, BasicDistributedZk2Test, SuggesterFSTTest, TestDeleteCollectionOnDownNodes, ActionThrottleTest, DistributedVersionInfoTest, TestSchemaSimilarityResource, MigrateRouteKeyTest, MinimalSchemaTest, HdfsDirectoryTest, TestSimpleQParserPlugin, SolrIndexMetricsTest, TestUtils, SolrCLIZkUtilsTest, TestSolrCloudWithSecureImpersonation, TestCollectionAPIs, TestDFRSimilarityFactory, TestSuggestSpellingConverter, TestPseudoReturnFields, PathHierarchyTokenizerFactoryTest, SolrInfoBeanTest, DocExpirationUpdateProcessorFactoryTest, BadIndexSchemaTest, MoveReplicaHDFSUlogDirTest, TestSolrCoreProperties, TestPayloadCheckQParserPlugin, HardAutoCommitTest, TestLRUStatsCache, FullHLLTest, TestGeoJSONResponseWriter, FullSolrCloudDistribCmdsTest, TestFoldingMultitermQuery, TestSizeLimitedDistributedMap, TestManagedResourceStorage, AutoCommitTest, TestLegacyNumericRangeQueryBuilder, LeaderElectionTest, TestTolerantUpdateProcessorRandomCloud, HdfsChaosMonkeyNothingIsSafeTest, VMParamsZkACLAndCredentialsProvidersTest, TestGroupingSearch, SortByFunctionTest, TestLFUCache, DeleteLastCustomShardedReplicaTest, TermVectorComponentDistributedTest, TestUninvertingReader, TestImplicitCoreProperties, DistributedFacetPivotLargeTest, UtilsToolTest, NotRequiredUniqueKeyTest, TestMacros, TestLegacyField, TestSSLRandomization, CloudExitableDirectoryReaderTest, TestLegacyFieldCache, TestSolrConfigHandlerCloud, DistributedDebugComponentTest, TestStressLiveNodes, PingRequestHandlerTest, MultiTermTest, UpdateLogTest, TestEmbeddedSolrServerSchemaAPI, NodeMutatorTest, ReplicationFactorTest, PrimitiveFieldTypeTest, SearchHandlerTest, TestPolicyCloud, DateFieldTest, TestRecovery, DocValuesMissingTest, TestCollapseQParserPlugin, HighlighterMaxOffsetTest, TestCollectionAPI, TestAuthenticationFramework, CircularListTest, TestSchemaManager, TestRTimerTree, SolrMetricManagerTest, DocumentBuilderTest, TestFieldCollectionResource, StressHdfsTest, TestRandomCollapseQParserPlugin, TestBulkSchemaConcurrent, TestSimpleTextCodec, TimeZoneUtilsTest, SignatureUpdateProcessorFactoryTest, SolrShardReporterTest, CdcrBootstrapTest, LeaderFailureAfterFreshStartTest, DistributedQueryComponentOptimizationTest, OverseerCollectionConfigSetProcessorTest, AtomicUpdateProcessorFactoryTest, CachingDirectoryFactoryTest, TestSolrQueryResponse, TestCodecSupport, TestFastWriter, TestGraphTermsQParserPlugin, TestReversedWildcardFilterFactory, NumericFieldsTest, TestRandomFlRTGCloud, SegmentsInfoRequestHandlerTest, TestZkChroot, SimpleFacetsTest, DateMathParserTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, ShardRoutingTest, ShardRoutingCustomTest, TestDistributedSearch]
   [junit4] Completed [380/731 (6!)] on J2 in 136.87s, 1 test, 1 error <<< FAILURES!

[...truncated 47386 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 156 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/156/

2 tests failed.
FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testPoissonDistribution

Error Message:
expected:<100.1277> but was:<96.83387609760834>

Stack Trace:
java.lang.AssertionError: expected:<100.1277> but was:<96.83387609760834>
        at __randomizedtesting.SeedInfo.seed([F844E3D2DEC14670:60541C737C783E0C]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:443)
        at org.junit.Assert.assertEquals(Assert.java:512)
        at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testPoissonDistribution(StreamExpressionTest.java:6318)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.LeaderElectionContextKeyTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([DF2AD6B9E6AE9553:577EE9634852F8AB]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11210 lines...]
   [junit4] Suite: org.apache.solr.cloud.LeaderElectionContextKeyTest
   [junit4]   2> 1    INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/init-core-data-001
   [junit4]   2> 1031 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1188 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.e.j.u.log Logging initialized @11531ms
   [junit4]   2> 1215 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1280 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/tempDir-001
   [junit4]   2> 1320 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1341 INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1342 INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1705 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:46145
   [junit4]   2> 2271 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 2326 ERROR (Thread-1) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4429 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e7826221d0000, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 5896 INFO  (jetty-launcher-1-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7357 INFO  (jetty-launcher-1-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d2869d8{/solr,null,AVAILABLE}
   [junit4]   2> 7405 INFO  (jetty-launcher-1-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@45d4b1f1{HTTP/1.1,[http/1.1]}{127.0.0.1:46645}
   [junit4]   2> 7406 INFO  (jetty-launcher-1-thread-1) [    ] o.e.j.s.Server Started @17749ms
   [junit4]   2> 7406 INFO  (jetty-launcher-1-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46645}
   [junit4]   2> 8378 ERROR (jetty-launcher-1-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 8378 INFO  (jetty-launcher-1-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 8391 INFO  (jetty-launcher-1-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 8391 INFO  (jetty-launcher-1-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 8752 INFO  (jetty-launcher-1-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-12T22:12:05.250Z
   [junit4]   2> 8913 INFO  (jetty-launcher-1-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 9176 INFO  (jetty-launcher-1-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 11922 INFO  (jetty-launcher-1-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46145/solr
   [junit4]   2> 14301 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:46645_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 14303 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:46645_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46645_solr
   [junit4]   2> 14324 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:46645_solr    ] o.a.s.c.Overseer Overseer (id=98648347025145859-127.0.0.1:46645_solr-n_0000000000) starting
   [junit4]   2> 15062 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:46645_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46645_solr
   [junit4]   2> 15070 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 18270 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:46645_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46645.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4277938e
   [junit4]   2> 18272 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:46645_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46645.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4277938e
   [junit4]   2> 18272 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:46645_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46645.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4277938e
   [junit4]   2> 19830 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:46645_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/tempDir-001/node1/.
   [junit4]   2> 22151 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 22154 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46145/solr ready
   [junit4]   2> 25493 INFO  (qtp763153725-26) [n:127.0.0.1:46645_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 25663 INFO  (OverseerThreadFactory-6-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.CreateCollectionCmd Create collection testCollection1
   [junit4]   2> 25837 INFO  (OverseerStateUpdate-98648347025145859-127.0.0.1:46645_solr-n_0000000000) [n:127.0.0.1:46645_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testCollection1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:46645/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 25849 INFO  (OverseerStateUpdate-98648347025145859-127.0.0.1:46645_solr-n_0000000000) [n:127.0.0.1:46645_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testCollection1_shard2_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:46645/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 26173 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 26182 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 26201 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 26332 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 26347 INFO  (zkCallback-8-thread-2-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 27309 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 27326 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 28944 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.IndexSchema [testCollection1_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 29019 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.IndexSchema [testCollection1_shard2_replica_n3] Schema name=minimal
   [junit4]   2> 30310 WARN  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 30312 WARN  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 30344 WARN  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 30351 WARN  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 30397 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 30399 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 30402 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'testCollection1_shard2_replica_n3' using configuration from collection testCollection1, trusted=true
   [junit4]   2> 30410 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCollection1_shard1_replica_n1' using configuration from collection testCollection1, trusted=true
   [junit4]   2> 30469 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46645.solr.core.testCollection1.shard2.replica_n3' (registry 'solr.core.testCollection1.shard2.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4277938e
   [junit4]   2> 30504 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46645.solr.core.testCollection1.shard1.replica_n1' (registry 'solr.core.testCollection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4277938e
   [junit4]   2> 30524 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 30526 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 30637 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore [[testCollection1_shard2_replica_n3] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/tempDir-001/node1/testCollection1_shard2_replica_n3], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/tempDir-001/node1/./testCollection1_shard2_replica_n3/data/]
   [junit4]   2> 30642 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore [[testCollection1_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/tempDir-001/node1/testCollection1_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/tempDir-001/node1/./testCollection1_shard1_replica_n1/data/]
   [junit4]   2> 33424 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 33424 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 33424 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 33433 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 33487 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 33488 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 33493 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 33502 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 33556 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2952e0dd[testCollection1_shard1_replica_n1] main]
   [junit4]   2> 33562 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.SolrIndexSearcher Opening [Searcher@42277ef4[testCollection1_shard2_replica_n3] main]
   [junit4]   2> 33608 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 33610 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 33655 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 33662 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 33682 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 33684 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 34031 INFO  (searcherExecutor-9-thread-1-processing-n:127.0.0.1:46645_solr x:testCollection1_shard2_replica_n3 s:shard2 c:testCollection1 r:core_node4) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SolrCore [testCollection1_shard2_replica_n3] Registered new searcher Searcher@42277ef4[testCollection1_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 34034 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578373586240929792
   [junit4]   2> 34052 INFO  (searcherExecutor-10-thread-1-processing-n:127.0.0.1:46645_solr x:testCollection1_shard1_replica_n1 s:shard1 c:testCollection1 r:core_node2) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore [testCollection1_shard1_replica_n1] Registered new searcher Searcher@2952e0dd[testCollection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 34055 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578373586262949888
   [junit4]   2> 34391 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 34391 INFO  (zkCallback-8-thread-2-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 34398 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 34398 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 34398 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46645/solr/testCollection1_shard1_replica_n1/
   [junit4]   2> 34399 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 34399 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:46645/solr/testCollection1_shard1_replica_n1/ has no replicas
   [junit4]   2> 34400 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 34435 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 34435 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 34435 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46645/solr/testCollection1_shard2_replica_n3/
   [junit4]   2> 34437 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 34437 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.SyncStrategy http://127.0.0.1:46645/solr/testCollection1_shard2_replica_n3/ has no replicas
   [junit4]   2> 34437 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 34485 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46645/solr/testCollection1_shard1_replica_n1/ shard1
   [junit4]   2> 34521 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46645/solr/testCollection1_shard2_replica_n3/ shard2
   [junit4]   2> 34626 INFO  (zkCallback-8-thread-2-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 34627 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 34667 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 34675 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=testCollection1_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=8506
   [junit4]   2> 34678 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 34685 INFO  (qtp763153725-29) [n:127.0.0.1:46645_solr c:testCollection1 s:shard2 r:core_node4 x:testCollection1_shard2_replica_n3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection1_shard2_replica_n3&action=CREATE&numShards=2&collection=testCollection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=8484
   [junit4]   2> 34794 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 34794 INFO  (zkCallback-8-thread-2-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 34798 INFO  (qtp763153725-26) [n:127.0.0.1:46645_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 35755 INFO  (OverseerCollectionConfigSetProcessor-98648347025145859-127.0.0.1:46645_solr-n_0000000000) [n:127.0.0.1:46645_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 35798 INFO  (qtp763153725-26) [n:127.0.0.1:46645_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=10486
   [junit4]   2> 35802 INFO  (qtp763153725-27) [n:127.0.0.1:46645_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 36965 INFO  (OverseerThreadFactory-6-thread-2-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.CreateCollectionCmd Create collection testCollection2
   [junit4]   2> 37290 INFO  (OverseerStateUpdate-98648347025145859-127.0.0.1:46645_solr-n_0000000000) [n:127.0.0.1:46645_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection2",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testCollection2_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:46645/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 37297 INFO  (OverseerStateUpdate-98648347025145859-127.0.0.1:46645_solr-n_0000000000) [n:127.0.0.1:46645_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testCollection2",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testCollection2_shard2_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:46645/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 37500 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 37502 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 37696 INFO  (zkCallback-8-thread-3-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 37696 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 38809 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 38827 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 38990 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.IndexSchema [testCollection2_shard2_replica_n2] Schema name=minimal
   [junit4]   2> 39014 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.IndexSchema [testCollection2_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 39061 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 39061 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testCollection2_shard2_replica_n2' using configuration from collection testCollection2, trusted=true
   [junit4]   2> 39062 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46645.solr.core.testCollection2.shard2.replica_n2' (registry 'solr.core.testCollection2.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4277938e
   [junit4]   2> 39062 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 39063 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore [[testCollection2_shard2_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/tempDir-001/node1/testCollection2_shard2_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/tempDir-001/node1/./testCollection2_shard2_replica_n2/data/]
   [junit4]   2> 39067 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 39068 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testCollection2_shard1_replica_n1' using configuration from collection testCollection2, trusted=true
   [junit4]   2> 39068 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46645.solr.core.testCollection2.shard1.replica_n1' (registry 'solr.core.testCollection2.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4277938e
   [junit4]   2> 39068 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 39068 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore [[testCollection2_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/tempDir-001/node1/testCollection2_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001/tempDir-001/node1/./testCollection2_shard1_replica_n1/data/]
   [junit4]   2> 39866 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 39866 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 39868 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 39868 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 39870 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@9f26fd9[testCollection2_shard1_replica_n1] main]
   [junit4]   2> 39872 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 39872 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 39873 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 39880 INFO  (searcherExecutor-20-thread-1-processing-n:127.0.0.1:46645_solr x:testCollection2_shard1_replica_n1 s:shard1 c:testCollection2 r:core_node3) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore [testCollection2_shard1_replica_n1] Registered new searcher Searcher@9f26fd9[testCollection2_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 39880 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578373592370905088
   [junit4]   2> 39889 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 39889 INFO  (zkCallback-8-thread-3-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 39891 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 39891 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 39891 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46645/solr/testCollection2_shard1_replica_n1/
   [junit4]   2> 39892 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 39892 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:46645/solr/testCollection2_shard1_replica_n1/ has no replicas
   [junit4]   2> 39892 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 39907 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46645/solr/testCollection2_shard1_replica_n1/ shard1
   [junit4]   2> 39938 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 39938 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 39940 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 39940 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 39989 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@e7436d2[testCollection2_shard2_replica_n2] main]
   [junit4]   2> 39990 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 39991 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 39992 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 39993 INFO  (searcherExecutor-19-thread-1-processing-n:127.0.0.1:46645_solr x:testCollection2_shard2_replica_n2 s:shard2 c:testCollection2 r:core_node4) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore [testCollection2_shard2_replica_n2] Registered new searcher Searcher@e7436d2[testCollection2_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 39994 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578373592490442752
   [junit4]   2> 40002 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 40002 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 40002 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46645/solr/testCollection2_shard2_replica_n2/
   [junit4]   2> 40002 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 40002 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:46645/solr/testCollection2_shard2_replica_n2/ has no replicas
   [junit4]   2> 40002 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 40026 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46645/solr/testCollection2_shard2_replica_n2/ shard2
   [junit4]   2> 40108 INFO  (zkCallback-8-thread-3-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 40108 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 40120 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 40122 INFO  (qtp763153725-28) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=testCollection2_shard1_replica_n1&action=CREATE&numShards=2&collection=testCollection2&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2620
   [junit4]   2> 40126 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 40128 INFO  (qtp763153725-30) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=config&newCollection=true&name=testCollection2_shard2_replica_n2&action=CREATE&numShards=2&collection=testCollection2&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2628
   [junit4]   2> 40132 INFO  (qtp763153725-27) [n:127.0.0.1:46645_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 40234 INFO  (zkCallback-8-thread-3-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 40234 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 40966 INFO  (OverseerCollectionConfigSetProcessor-98648347025145859-127.0.0.1:46645_solr-n_0000000000) [n:127.0.0.1:46645_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 41215 INFO  (qtp763153725-27) [n:127.0.0.1:46645_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=config&maxShardsPerNode=1000&name=testCollection2&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=5413
   [junit4]   2> 41222 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: testCollection1 failOnTimeout:true timeout (sec):30
   [junit4]   2> 41223 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: testCollection1
   [junit4]   2> 41223 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: testCollection2 failOnTimeout:true timeout (sec):30
   [junit4]   2> 41224 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: testCollection2
   [junit4]   2> 41258 INFO  (TEST-LeaderElectionContextKeyTest.test-seed#[DF2AD6B9E6AE9553]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 41284 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection1.shard2.replica_n3, tag=null
   [junit4]   2> 41285 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@22ebcdcd: rootName = solr_46645, domain = solr.core.testCollection1.shard2.replica_n3, service url = null, agent id = null] for registry solr.core.testCollection1.shard2.replica_n3 / com.codahale.metrics.MetricRegistry@6db45aff
   [junit4]   2> 41400 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr    ] o.a.s.c.SolrCore [testCollection1_shard2_replica_n3]  CLOSING SolrCore org.apache.solr.core.SolrCore@521d9170
   [junit4]   2> 41400 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection1.shard2.replica_n3, tag=1377669488
   [junit4]   2> 41400 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection1.shard2.leader, tag=1377669488
   [junit4]   2> 41439 INFO  (qtp763153725-24) [n:127.0.0.1:46645_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=false&core=testCollection1_shard2_replica_n3&deleteDataDir=false&action=UNLOAD&indexInfo=true&wt=javabin&version=2&deleteIndex=false} status=0 QTime=155
   [junit4]   2> 41442 INFO  (TEST-LeaderElectionContextKeyTest.test-seed#[DF2AD6B9E6AE9553]) [    ] o.a.s.SolrTestCaseJ4 ###Ending test
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LeaderElectionContextKeyTest -Dtests.method=test -Dtests.seed=DF2AD6B9E6AE9553 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=el -Dtests.timezone=America/Mexico_City -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.22s J0 | LeaderElectionContextKeyTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > at __randomizedtesting.SeedInfo.seed([DF2AD6B9E6AE9553:577EE9634852F8AB]:0)
   [junit4]    > at org.apache.solr.cloud.LeaderElectionContextKeyTest.test(LeaderElectionContextKeyTest.java:88)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 41476 INFO  (jetty-closer-2-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@45d4b1f1{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 41476 INFO  (jetty-closer-2-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1254655134
   [junit4]   2> 41477 INFO  (jetty-closer-2-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 41477 INFO  (jetty-closer-2-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@45654465: rootName = solr_46645, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@351b7807
   [junit4]   2> 41511 INFO  (jetty-closer-2-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 41511 INFO  (jetty-closer-2-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@79b2d4cc: rootName = solr_46645, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5ab55427
   [junit4]   2> 41539 INFO  (zkCallback-8-thread-3-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 41539 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 41558 INFO  (jetty-closer-2-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 41558 INFO  (jetty-closer-2-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@756c1b5a: rootName = solr_46645, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@1cbe037d
   [junit4]   2> 41586 INFO  (jetty-closer-2-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 41791 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 41792 INFO  (zkCallback-8-thread-3-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection1/state.json] for collection [testCollection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 41796 INFO  (zkCallback-8-thread-3-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 41796 INFO  (zkCallback-8-thread-1-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCollection2/state.json] for collection [testCollection2] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 41817 INFO  (coreCloseExecutor-29-thread-1) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.c.SolrCore [testCollection1_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@70f89bb9
   [junit4]   2> 41817 INFO  (coreCloseExecutor-29-thread-1) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection1.shard1.replica_n1, tag=1895340985
   [junit4]   2> 41817 INFO  (coreCloseExecutor-29-thread-1) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5c87468c: rootName = solr_46645, domain = solr.core.testCollection1.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.testCollection1.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@544403d5
   [junit4]   2> 41841 INFO  (coreCloseExecutor-29-thread-2) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.c.SolrCore [testCollection2_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@27eaa8f7
   [junit4]   2> 41842 INFO  (coreCloseExecutor-29-thread-3) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.c.SolrCore [testCollection2_shard2_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@3a08d9e2
   [junit4]   2> 41847 INFO  (coreCloseExecutor-29-thread-3) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection2.shard2.replica_n2, tag=973658594
   [junit4]   2> 41847 INFO  (coreCloseExecutor-29-thread-3) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2fc9b9fa: rootName = solr_46645, domain = solr.core.testCollection2.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.testCollection2.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@1b789bb4
   [junit4]   2> 41882 INFO  (coreCloseExecutor-29-thread-2) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testCollection2.shard1.replica_n1, tag=669690103
   [junit4]   2> 41882 INFO  (coreCloseExecutor-29-thread-2) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@53f7d6a6: rootName = solr_46645, domain = solr.core.testCollection2.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.testCollection2.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@51f4572c
   [junit4]   2> 41926 INFO  (coreCloseExecutor-29-thread-1) [n:127.0.0.1:46645_solr c:testCollection1 s:shard1 r:core_node2 x:testCollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection1.shard1.leader, tag=1895340985
   [junit4]   2> 41934 INFO  (coreCloseExecutor-29-thread-3) [n:127.0.0.1:46645_solr c:testCollection2 s:shard2 r:core_node4 x:testCollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection2.shard2.leader, tag=973658594
   [junit4]   2> 41941 INFO  (coreCloseExecutor-29-thread-2) [n:127.0.0.1:46645_solr c:testCollection2 s:shard1 r:core_node3 x:testCollection2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testCollection2.shard1.leader, tag=669690103
   [junit4]   2> 41952 INFO  (jetty-closer-2-thread-1) [    ] o.a.s.c.Overseer Overseer (id=98648347025145859-127.0.0.1:46645_solr-n_0000000000) closing
   [junit4]   2> 41953 INFO  (OverseerStateUpdate-98648347025145859-127.0.0.1:46645_solr-n_0000000000) [n:127.0.0.1:46645_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:46645_solr
   [junit4]   2> 43465 WARN  (zkCallback-8-thread-3-processing-n:127.0.0.1:46645_solr) [n:127.0.0.1:46645_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 43467 INFO  (jetty-closer-2-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6d2869d8{/solr,null,UNAVAILABLE}
   [junit4]   2> 43470 ERROR (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 43471 INFO  (SUITE-LeaderElectionContextKeyTest-seed#[DF2AD6B9E6AE9553]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46145 46145
   [junit4]   2> 43793 INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46145 46145
   [junit4]   2> 43796 WARN  (Thread-1) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 2 /solr/aliases.json
   [junit4]   2> 2 /solr/clusterprops.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 8 /solr/collections/testCollection1/state.json
   [junit4]   2> 7 /solr/collections/testCollection2/state.json
   [junit4]   2> 2 /solr/clusterstate.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 2 /solr/live_nodes
   [junit4]   2> 2 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderElectionContextKeyTest_DF2AD6B9E6AE9553-001
   [junit4]   2> Sep 12, 2017 10:12:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=873, maxMBSortInHeap=5.6230208906220245, sim=RandomSimilarity(queryNorm=false): {}, locale=el, timezone=America/Mexico_City
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=273837344,total=310378496
   [junit4]   2> NOTE: All tests run in this JVM: [LeaderElectionContextKeyTest]
   [junit4] Completed [2/731 (1!)] on J0 in 51.27s, 1 test, 1 failure <<< FAILURES!

[...truncated 3428 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.io.stream.StreamExpressionTest
   [junit4]   2> 325904 INFO  (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_F844E3D2DEC14670-001/init-core-data-001
   [junit4]   2> 325919 WARN  (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=35 numCloses=35
   [junit4]   2> 325919 INFO  (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 325920 INFO  (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 325933 INFO  (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_F844E3D2DEC14670-001/tempDir-001
   [junit4]   2> 325933 INFO  (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 325935 INFO  (Thread-179) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 325935 INFO  (Thread-179) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 325962 ERROR (Thread-179) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 326045 INFO  (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:60440
   [junit4]   2> 326077 INFO  (jetty-launcher-310-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 326080 INFO  (jetty-launcher-310-thread-3) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 326082 INFO  (jetty-launcher-310-thread-4) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 326086 INFO  (jetty-launcher-310-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 326094 INFO  (jetty-launcher-310-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@41048730{/solr,null,AVAILABLE}
   [junit4]   2> 326105 INFO  (jetty-launcher-310-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1e8b351b{/solr,null,AVAILABLE}
   [junit4]   2> 326105 INFO  (jetty-launcher-310-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@70916410{SSL,[ssl, http/1.1]}{127.0.0.1:49858}
   [junit4]   2> 326105 INFO  (jetty-launcher-310-thread-1) [    ] o.e.j.s.Server Started @335301ms
   [junit4]   2> 326105 INFO  (jetty-launcher-310-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49858}
   [junit4]   2> 326106 ERROR (jetty-launcher-310-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 326106 INFO  (jetty-launcher-310-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 326106 INFO  (jetty-launcher-310-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 326106 INFO  (jetty-launcher-310-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 326106 INFO  (jetty-launcher-310-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-12T23:41:43.303Z
   [junit4]   2> 326119 INFO  (jetty-launcher-310-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4058f2c2{/solr,null,AVAILABLE}
   [junit4]   2> 326119 INFO  (jetty-launcher-310-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@29a46b73{SSL,[ssl, http/1.1]}{127.0.0.1:56938}
   [junit4]   2> 326119 INFO  (jetty-launcher-310-thread-4) [    ] o.e.j.s.Server Started @335315ms
   [junit4]   2> 326119 INFO  (jetty-launcher-310-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56938}
   [junit4]   2> 326119 ERROR (jetty-launcher-310-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 326119 INFO  (jetty-launcher-310-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 326119 INFO  (jetty-launcher-310-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 326119 INFO  (jetty-launcher-310-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 326119 INFO  (jetty-launcher-310-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-12T23:41:43.316Z
   [junit4]   2> 326127 INFO  (jetty-launcher-310-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5654372b{/solr,null,AVAILABLE}
   [junit4]   2> 326130 INFO  (jetty-launcher-310-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@10b2f380{SSL,[ssl, http/1.1]}{127.0.0.1:40368}
   [junit4]   2> 326130 INFO  (jetty-launcher-310-thread-3) [    ] o.e.j.s.Server Started @335326ms
   [junit4]   2> 326130 INFO  (jetty-launcher-310-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40368}
   [junit4]   2> 326130 ERROR (jetty-launcher-310-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 326130 INFO  (jetty-launcher-310-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 326130 INFO  (jetty-launcher-310-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 326130 INFO  (jetty-launcher-310-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 326130 INFO  (jetty-launcher-310-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-12T23:41:43.327Z
   [junit4]   2> 326138 INFO  (jetty-launcher-310-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2bb9307c{SSL,[ssl, http/1.1]}{127.0.0.1:51938}
   [junit4]   2> 326138 INFO  (jetty-launcher-310-thread-2) [    ] o.e.j.s.Server Started @335334ms
   [junit4]   2> 326138 INFO  (jetty-launcher-310-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51938}
   [junit4]   2> 326139 ERROR (jetty-launcher-310-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 326139 INFO  (jetty-launcher-310-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 326139 INFO  (jetty-launcher-310-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 326139 INFO  (jetty-launcher-310-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 326139 INFO  (jetty-launcher-310-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-12T23:41:43.336Z
   [junit4]   2> 326156 INFO  (jetty-launcher-310-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 326161 INFO  (jetty-launcher-310-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 326166 INFO  (jetty-launcher-310-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 326170 INFO  (jetty-launcher-310-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 326204 INFO  (jetty-launcher-310-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60440/solr
   [junit4]   2> 326220 INFO  (jetty-launcher-310-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60440/solr
   [junit4]   2> 326227 INFO  (jetty-launcher-310-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60440/solr
   [junit4]   2> 326229 INFO  (jetty-launcher-310-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60440/solr
   [junit4]   2> 326302 WARN  (jetty-launcher-310-thread-2) [n:127.0.0.1:51938_solr    ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
   [junit4]   2> 326404 WARN  (jetty-launcher-310-thread-3) [n:127.0.0.1:40368_solr    ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
   [junit4]   2> 326378 WARN  (jetty-launcher-310-thread-4) [n:127.0.0.1:56938_solr    ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
   [junit4]   2> 326312 WARN  (jetty-launcher-310-thread-1) [n:127.0.0.1:49858_solr    ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
   [junit4]   2> 326410 INFO  (jetty-launcher-310-thread-2) [n:127.0.0.1:51938_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 326417 INFO  (jetty-launcher-310-thread-4) [n:127.0.0.1:56938_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 326417 INFO  (jetty-launcher-310-thread-4) [n:127.0.0.1:56938_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56938_solr
   [junit4]   2> 326418 INFO  (jetty-launcher-310-thread-4) [n:127.0.0.1:56938_solr    ] o.a.s.c.Overseer Overseer (id=98648699904589836-127.0.0.1:56938_solr-n_0000000000) starting
   [junit4]   2> 326423 INFO  (jetty-launcher-310-thread-1) [n:127.0.0.1:49858_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 326423 INFO  (jetty-launcher-310-thread-1) [n:127.0.0.1:49858_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49858_solr
   [junit4]   2> 326436 INFO  (jetty-launcher-310-thread-2) [n:127.0.0.1:51938_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51938_solr
   [junit4]   2> 326451 INFO  (jetty-launcher-310-thread-3) [n:127.0.0.1:40368_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 326453 INFO  (jetty-launcher-310-thread-3) [n:127.0.0.1:40368_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 326454 INFO  (jetty-launcher-310-thread-3) [n:127.0.0.1:40368_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40368_solr
   [junit4]   2> 326465 INFO  (jetty-launcher-310-thread-4) [n:127.0.0.1:56938_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56938_solr
   [junit4]   2> 326511 INFO  (zkCallback-331-thread-1-processing-n:127.0.0.1:49858_solr) [n:127.0.0.1:49858_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 326511 INFO  (zkCallback-329-thread-1-processing-n:127.0.0.1:51938_solr) [n:127.0.0.1:51938_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 326511 INFO  (zkCallback-332-thread-1-processing-n:127.0.0.1:56938_solr) [n:127.0.0.1:56938_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 326511 INFO  (zkCallback-330-thread-1-processing-n:127.0.0.1:40368_solr) [n:127.0.0.1:40368_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (4)
   [junit4]   2> 327529 INFO  (jetty-launcher-310-thread-4) [n:127.0.0.1:56938_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56938.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 327722 INFO  (jetty-launcher-310-thread-4) [n:127.0.0.1:56938_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56938.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 327723 INFO  (jetty-launcher-310-thread-4) [n:127.0.0.1:56938_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56938.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 327724 INFO  (jetty-launcher-310-thread-4) [n:127.0.0.1:56938_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_F844E3D2DEC14670-001/tempDir-001/node4/.
   [junit4]   2> 327869 INFO  (jetty-launcher-310-thread-3) [n:127.0.0.1:40368_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40368.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 328062 INFO  (jetty-launcher-310-thread-3) [n:127.0.0.1:40368_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40368.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 328062 INFO  (jetty-launcher-310-thread-3) [n:127.0.0.1:40368_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40368.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 328066 INFO  (jetty-launcher-310-thread-3) [n:127.0.0.1:40368_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_F844E3D2DEC14670-001/tempDir-001/node3/.
   [junit4]   2> 328258 INFO  (jetty-launcher-310-thread-2) [n:127.0.0.1:51938_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51938.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 328389 INFO  (jetty-launcher-310-thread-2) [n:127.0.0.1:51938_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51938.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 328389 INFO  (jetty-launcher-310-thread-2) [n:127.0.0.1:51938_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51938.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 328390 INFO  (jetty-launcher-310-thread-2) [n:127.0.0.1:51938_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_F844E3D2DEC14670-001/tempDir-001/node2/.
   [junit4]   2> 328515 INFO  (jetty-launcher-310-thread-1) [n:127.0.0.1:49858_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49858.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 328578 INFO  (jetty-launcher-310-thread-1) [n:127.0.0.1:49858_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49858.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 328578 INFO  (jetty-launcher-310-thread-1) [n:127.0.0.1:49858_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49858.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@66dd2ce7
   [junit4]   2> 328771 INFO  (jetty-launcher-310-thread-1) [n:127.0.0.1:49858_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_F844E3D2DEC14670-001/tempDir-001/node1/.
   [junit4]   2> 329132 INFO  (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 329132 INFO  (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60440/solr ready
   [junit4]   2> 329160 INFO  (qtp1142243660-1396) [n:127.0.0.1:51938_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 329175 INFO  (OverseerThreadFactory-438-thread-1-processing-n:127.0.0.1:56938_solr) [n:127.0.0.1:56938_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 329285 INFO  (OverseerStateUpdate-98648699904589836-127.0.0.1:56938_solr-n_0000000000) [n:127.0.0.1:56938_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:40368/solr",
   [junit4]   2>   "type":"NRT"}
   [junit4]   2> 329287 INFO  (OverseerStateUpdate-98648699904589836-127.0.0.1:56938_solr-n_0000000000) [n:127.0.0.1:56938_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"collection1_shard2_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:56938/solr",
   [junit4]   2>   "ty

[...truncated too long message...]

collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard2.replica_n2, tag=328981916
   [junit4]   2> 521432 INFO  (coreCloseExecutor-637-thread-1) [n:127.0.0.1:56938_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6bc423d9: rootName = solr_56938, domain = solr.core.collection1.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.collection1.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@1b2b187c
   [junit4]   2> 521435 INFO  (jetty-closer-311-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 521449 INFO  (jetty-closer-311-thread-3) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 521459 INFO  (coreCloseExecutor-638-thread-1) [n:127.0.0.1:40368_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7f23379e
   [junit4]   2> 521462 INFO  (jetty-closer-311-thread-4) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 521462 INFO  (jetty-closer-311-thread-4) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7a7b7044: rootName = solr_49858, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@3300c434
   [junit4]   2> 521462 INFO  (coreCloseExecutor-638-thread-1) [n:127.0.0.1:40368_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard1.replica_n1, tag=2133014430
   [junit4]   2> 521462 INFO  (coreCloseExecutor-638-thread-1) [n:127.0.0.1:40368_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3e7c1116: rootName = solr_40368, domain = solr.core.collection1.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.collection1.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@4cb07697
   [junit4]   2> 521467 INFO  (coreCloseExecutor-637-thread-1) [n:127.0.0.1:56938_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=328981916
   [junit4]   2> 521497 INFO  (zkCallback-332-thread-1-processing-n:127.0.0.1:56938_solr) [n:127.0.0.1:56938_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 521497 INFO  (zkCallback-330-thread-2-processing-n:127.0.0.1:40368_solr) [n:127.0.0.1:40368_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 521497 INFO  (zkCallback-331-thread-1-processing-n:127.0.0.1:49858_solr) [n:127.0.0.1:49858_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 521499 INFO  (jetty-closer-311-thread-4) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 521500 INFO  (coreCloseExecutor-638-thread-1) [n:127.0.0.1:40368_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=2133014430
   [junit4]   2> 521515 WARN  (zkCallback-331-thread-1-processing-n:127.0.0.1:49858_solr) [n:127.0.0.1:49858_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 521515 INFO  (zkCallback-332-thread-1-processing-n:127.0.0.1:56938_solr) [n:127.0.0.1:56938_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 521515 INFO  (zkCallback-330-thread-1-processing-n:127.0.0.1:40368_solr) [n:127.0.0.1:40368_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 521520 INFO  (jetty-closer-311-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@41048730{/solr,null,UNAVAILABLE}
   [junit4]   2> 521522 INFO  (jetty-closer-311-thread-1) [    ] o.a.s.c.Overseer Overseer (id=98648699904589836-127.0.0.1:56938_solr-n_0000000000) closing
   [junit4]   2> 521522 INFO  (OverseerStateUpdate-98648699904589836-127.0.0.1:56938_solr-n_0000000000) [n:127.0.0.1:56938_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:56938_solr
   [junit4]   2> 521528 INFO  (zkCallback-332-thread-1-processing-n:127.0.0.1:56938_solr) [n:127.0.0.1:56938_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 522993 WARN  (zkCallback-329-thread-2-processing-n:127.0.0.1:51938_solr) [n:127.0.0.1:51938_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 522993 INFO  (jetty-closer-311-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4058f2c2{/solr,null,UNAVAILABLE}
   [junit4]   2> 523028 WARN  (zkCallback-330-thread-2-processing-n:127.0.0.1:40368_solr) [n:127.0.0.1:40368_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 523028 INFO  (jetty-closer-311-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5654372b{/solr,null,UNAVAILABLE}
   [junit4]   2> 523038 WARN  (zkCallback-332-thread-1-processing-n:127.0.0.1:56938_solr) [n:127.0.0.1:56938_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 523038 INFO  (jetty-closer-311-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1e8b351b{/solr,null,UNAVAILABLE}
   [junit4]   2> 523039 ERROR (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 523039 INFO  (SUITE-StreamExpressionTest-seed#[F844E3D2DEC14670]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60440 60440
   [junit4]   2> 523207 INFO  (Thread-179) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60440 60440
   [junit4]   2> 523207 WARN  (Thread-179) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 93 /solr/aliases.json
   [junit4]   2> 11 /solr/configs/conf
   [junit4]   2> 8 /solr/configs/ml
   [junit4]   2> 4 /solr/security.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 93 /solr/clusterprops.json
   [junit4]   2> 93 /solr/clusterstate.json
   [junit4]   2> 42 /solr/collections/destinationCollection/state.json
   [junit4]   2> 35 /solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 22 /solr/collections/destination/state.json
   [junit4]   2> 22 /solr/collections/mainCorpus/state.json
   [junit4]   2> 20 /solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 20 /solr/collections/workQueue/state.json
   [junit4]   2> 17 /solr/collections/collection1/state.json
   [junit4]   2> 10 /solr/collections/uknownCollection/state.json
   [junit4]   2> 10 /solr/collections/modelCollection/state.json
   [junit4]   2> 10 /solr/collections/checkpointCollection/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/98648699904589836-127.0.0.1:56938_solr-n_0000000000
   [junit4]   2> 2 /solr/overseer_elect/election/98648699904589835-127.0.0.1:49858_solr-n_0000000001
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 93 /solr/live_nodes
   [junit4]   2> 93 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_F844E3D2DEC14670-001
   [junit4]   2> Sep 12, 2017 11:45:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=lt, timezone=Africa/Khartoum
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=304591272,total=522190848
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrProperties, TestValidatingJsonMap, RegressionEvaluatorTest, TestDocumentObjectBinder, QueryResponseTest, TestUpdateRequestCodec, SolrExampleEmbeddedTest, CosineEvaluatorTest, TestCloudSolrClientConnections, TestCollectionStateWatchers, ClientUtilsTest, SolrDocumentTest, TangentEvaluatorTest, NormalizeEvaluatorTest, CloudSolrClientTest, SolrExampleStreamingBinaryTest, FacetFieldTest, GreaterThanEqualToEvaluatorTest, TestSuggesterResponse, HyperbolicSineEvaluatorTest, GetByIdTest, PowerEvaluatorTest, TestCollectionAdminRequest, TestPolicy, CloudSolrClientBuilderTest, CloudSolrClientCacheTest, ConcurrentUpdateSolrClientTest, HttpSolrClientBuilderTest, LBHttpSolrClientBuilderTest, GraphExpressionTest, StreamExpressionTest]
   [junit4] Completed [145/145 (1!)] on J1 in 198.28s, 109 tests, 1 failure <<< FAILURES!

[...truncated 45318 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 157 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/157/

8 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([46CBDB43005D3CE8:CE9FE499AEA15110]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ZkStateReader.forceUpdateCollection(ZkStateReader.java:323)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.updateMappingsFromZk(AbstractFullDistribZkTestBase.java:674)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.updateMappingsFromZk(AbstractFullDistribZkTestBase.java:669)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryAndCompareShards(AbstractFullDistribZkTestBase.java:1133)
        at org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:165)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 47 more


FAILED:  org.apache.solr.cloud.CollectionsAPISolrJTest.testSplitShard

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:45882/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:45882/solr
        at __randomizedtesting.SeedInfo.seed([46CBDB43005D3CE8:9DC1762F1EA80057]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:638)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.CollectionsAPISolrJTest.testSplitShard(CollectionsAPISolrJTest.java:243)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:526)
        ... 49 more


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:49268/collection2]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:49268/collection2]
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:665)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:49268/collection2: no servers hosting shard: shard4
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        ... 49 more


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node5:{"core":"c8n_1x3_lf_shard1_replica_n1","base_url":"http://127.0.0.1:45025","node_name":"127.0.0.1:45025_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/17)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node4":{           "core":"c8n_1x3_lf_shard1_replica_n2",           "base_url":"http://127.0.0.1:52300",           "node_name":"127.0.0.1:52300_",           "state":"down",           "type":"NRT"},         "core_node5":{           "core":"c8n_1x3_lf_shard1_replica_n1",           "base_url":"http://127.0.0.1:45025",           "node_name":"127.0.0.1:45025_",           "state":"active",           "type":"NRT",           "leader":"true"},         "core_node6":{           "state":"down",           "base_url":"http://127.0.0.1:55415",           "core":"c8n_1x3_lf_shard1_replica_n3",           "node_name":"127.0.0.1:55415_",           "type":"NRT"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"3",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node5:{"core":"c8n_1x3_lf_shard1_replica_n1","base_url":"http://127.0.0.1:45025","node_name":"127.0.0.1:45025_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/17)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"c8n_1x3_lf_shard1_replica_n2",
          "base_url":"http://127.0.0.1:52300",
          "node_name":"127.0.0.1:52300_",
          "state":"down",
          "type":"NRT"},
        "core_node5":{
          "core":"c8n_1x3_lf_shard1_replica_n1",
          "base_url":"http://127.0.0.1:45025",
          "node_name":"127.0.0.1:45025_",
          "state":"active",
          "type":"NRT",
          "leader":"true"},
        "core_node6":{
          "state":"down",
          "base_url":"http://127.0.0.1:55415",
          "core":"c8n_1x3_lf_shard1_replica_n3",
          "node_name":"127.0.0.1:55415_",
          "type":"NRT"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"3",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([46CBDB43005D3CE8:CE9FE499AEA15110]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.test

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
        at __randomizedtesting.SeedInfo.seed([46CBDB43005D3CE8:CE9FE499AEA15110]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1115)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:128)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:108)
        at org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.waitTillNodesActive(LeaderFailureAfterFreshStartTest.java:208)
        at org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.restartNodes(LeaderFailureAfterFreshStartTest.java:173)
        at org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.test(LeaderFailureAfterFreshStartTest.java:148)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1127)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1113)
        ... 47 more


FAILED:  org.apache.solr.cloud.TestAuthenticationFramework.testBasics

Error Message:
Error from server at http://127.0.0.1:44237/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason: <pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:44237/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>

        at __randomizedtesting.SeedInfo.seed([46CBDB43005D3CE8:7B13756F38B36298]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:539)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at org.apache.solr.cloud.TestAuthenticationFramework.collectionCreateSearchDeleteTwice(TestAuthenticationFramework.java:126)
        at org.apache.solr.cloud.TestAuthenticationFramework.testBasics(TestAuthenticationFramework.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:44237/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre>    Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:591)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:516)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  org.apache.solr.cloud.TestHdfsCloudBackupRestore.test

Error Message:
Error from server at https://127.0.0.1:37860/solr: Could not restore core

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:37860/solr: Could not restore core
        at __randomizedtesting.SeedInfo.seed([46CBDB43005D3CE8:CE9FE499AEA15110]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:275)
        at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:42569/delete_data_dir

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:42569/delete_data_dir
        at __randomizedtesting.SeedInfo.seed([46CBDB43005D3CE8:CE9FE499AEA15110]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:638)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:213)
        at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:526)
        ... 53 more




Build Log:
[...truncated 11476 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> 864885 INFO  (SUITE-BasicDistributedZkTest-seed#[46CBDB43005D3CE8]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/init-core-data-001
   [junit4]   2> 864885 WARN  (SUITE-BasicDistributedZkTest-seed#[46CBDB43005D3CE8]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 864885 INFO  (SUITE-BasicDistributedZkTest-seed#[46CBDB43005D3CE8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 864887 INFO  (SUITE-BasicDistributedZkTest-seed#[46CBDB43005D3CE8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 864887 INFO  (SUITE-BasicDistributedZkTest-seed#[46CBDB43005D3CE8]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /ss/il
   [junit4]   2> 864940 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 864940 INFO  (Thread-410) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 864940 INFO  (Thread-410) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 866236 ERROR (Thread-410) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 866237 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.ZkTestServer start zk server on port:58308
   [junit4]   2> 866492 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 866497 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 866501 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 866504 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 866508 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 866512 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 866513 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 866516 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 866520 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 866521 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 866524 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 866532 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 868059 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 868149 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6712b3c0{/ss/il,null,AVAILABLE}
   [junit4]   2> 868153 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@39891ca5{HTTP/1.1,[http/1.1]}{127.0.0.1:46140}
   [junit4]   2> 868153 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.e.j.s.Server Started @890954ms
   [junit4]   2> 868153 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/tempDir-001/control/data, hostContext=/ss/il, hostPort=46140, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/control-001/cores}
   [junit4]   2> 868154 ERROR (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 868154 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 868154 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 868154 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 868154 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-14T19:12:56.882Z
   [junit4]   2> 868212 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 868212 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/control-001/solr.xml
   [junit4]   2> 868254 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5651c185, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 868258 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58308/solr
   [junit4]   2> 869144 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 869145 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46140_ss%2Fil
   [junit4]   2> 869145 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.Overseer Overseer (id=98658967467327492-127.0.0.1:46140_ss%2Fil-n_0000000000) starting
   [junit4]   2> 869280 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46140_ss%2Fil
   [junit4]   2> 869328 INFO  (OverseerStateUpdate-98658967467327492-127.0.0.1:46140_ss%2Fil-n_0000000000) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 870583 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5651c185
   [junit4]   2> 872671 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5651c185
   [junit4]   2> 872672 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5651c185
   [junit4]   2> 872673 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/control-001/cores
   [junit4]   2> 872868 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 872869 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58308/solr ready
   [junit4]   2> 872979 INFO  (qtp17138440-1830) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:46140_ss%252Fil&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 873001 INFO  (OverseerThreadFactory-795-thread-1-processing-n:127.0.0.1:46140_ss%2Fil) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 873188 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 873224 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 873379 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46140_ss%2Fil) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 874305 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 874397 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 875497 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 875710 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 875710 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5651c185
   [junit4]   2> 875710 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 875710 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 875745 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20774504153317946]
   [junit4]   2> 875916 WARN  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 876328 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 876328 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 876329 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 876329 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 876355 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=23.63671875, floorSegmentMB=1.9208984375, forceMergeDeletesPctAllowed=4.030745292615885, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.38705089376779866
   [junit4]   2> 876398 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1721785f[control_collection_shard1_replica_n1] main]
   [junit4]   2> 876399 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 876399 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 876400 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 876400 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578543491451977728
   [junit4]   2> 876403 INFO  (searcherExecutor-798-thread-1-processing-n:127.0.0.1:46140_ss%2Fil x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@1721785f[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 876460 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46140_ss%2Fil) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 876463 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 876463 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 876463 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46140/ss/il/control_collection_shard1_replica_n1/
   [junit4]   2> 876463 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 876463 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:46140/ss/il/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 876463 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 876467 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46140/ss/il/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 876576 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46140_ss%2Fil) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 876592 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 876595 INFO  (qtp17138440-1828) [n:127.0.0.1:46140_ss%2Fil c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3406
   [junit4]   2> 876701 INFO  (qtp17138440-1830) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 876701 INFO  (zkCallback-369-thread-1-processing-n:127.0.0.1:46140_ss%2Fil) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 877052 INFO  (OverseerCollectionConfigSetProcessor-98658967467327492-127.0.0.1:46140_ss%2Fil-n_0000000000) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 877821 INFO  (qtp17138440-1830) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:46140_ss%252Fil&wt=javabin&version=2} status=0 QTime=4842
   [junit4]   2> 877965 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 877965 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58308/solr ready
   [junit4]   2> 877965 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 877985 INFO  (qtp17138440-1831) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 877987 INFO  (OverseerThreadFactory-795-thread-2-processing-n:127.0.0.1:46140_ss%2Fil) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 878006 WARN  (OverseerThreadFactory-795-thread-2-processing-n:127.0.0.1:46140_ss%2Fil) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 878420 INFO  (qtp17138440-1831) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 878421 INFO  (qtp17138440-1831) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=436
   [junit4]   2> 879670 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/shard-1-001 of type NRT
   [junit4]   2> 879671 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 879673 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@408f3ec7{/ss/il,null,AVAILABLE}
   [junit4]   2> 879673 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@54ea4053{HTTP/1.1,[http/1.1]}{127.0.0.1:60933}
   [junit4]   2> 879673 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.e.j.s.Server Started @902474ms
   [junit4]   2> 879673 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/ss/il, hostPort=60933, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/shard-1-001/cores}
   [junit4]   2> 879673 ERROR (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 879673 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 879673 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 879673 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 879674 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-09-14T19:13:08.402Z
   [junit4]   2> 879808 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 879808 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/shard-1-001/solr.xml
   [junit4]   2> 879848 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5651c185, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 879872 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58308/solr
   [junit4]   2> 881397 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 881413 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 881421 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60933_ss%2Fil
   [junit4]   2> 881537 INFO  (zkCallback-376-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 881537 INFO  (zkCallback-381-thread-1-processing-n:127.0.0.1:60933_ss%2Fil) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 881588 INFO  (OverseerCollectionConfigSetProcessor-98658967467327492-127.0.0.1:46140_ss%2Fil-n_0000000000) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 881589 INFO  (zkCallback-369-thread-2-processing-n:127.0.0.1:46140_ss%2Fil) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 882894 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5651c185
   [junit4]   2> 882944 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5651c185
   [junit4]   2> 882944 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5651c185
   [junit4]   2> 883041 INFO  (TEST-BasicDistributedZkTest.test-seed#[46CBDB43005D3CE8]) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/shard-1-001/cores
   [junit4]   2> 884526 INFO  (qtp17138440-1827) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:60933_ss%252Fil&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 884557 INFO  (OverseerThreadFactory-795-thread-3-processing-n:127.0.0.1:46140_ss%2Fil) [n:127.0.0.1:46140_ss%2Fil    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:60933_ss%2Fil for creating new replica
   [junit4]   2> 884658 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 884659 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 884671 INFO  (zkCallback-381-thread-1-processing-n:127.0.0.1:60933_ss%2Fil) [n:127.0.0.1:60933_ss%2Fil    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 885788 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 885823 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
   [junit4]   2> 887462 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 887521 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
   [junit4]   2> 887530 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5651c185
   [junit4]   2> 887530 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 887530 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_46CBDB43005D3CE8-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
   [junit4]   2> 887544 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20774504153317946]
   [junit4]   2> 887569 WARN  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 888572 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 888572 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 888573 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 888573 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 888600 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=23.63671875, floorSegmentMB=1.9208984375, forceMergeDeletesPctAllowed=4.030745292615885, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.38705089376779866
   [junit4]   2> 888621 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@471cb768[collection1_shard2_replica_n41] main]
   [junit4]   2> 888622 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 888623 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 888624 INFO  (qtp855374969-1871) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 888645 INFO  (searcherExecutor-809-thread-1-processing-n:127.0.0.1:60933_ss%2Fil x:collection1_shard2_replica_n41 s:shard2 c:collection1) [n:127.0.0.1:60933_ss%2Fil c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@471cb768[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 888664 INFO  (qtp855374969-1871) [n:127.

[...truncated too long message...]

4 x:hdfsbackuprestore_restored_shard1_1_replica_n63] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.hdfsbackuprestore_restored.shard1_1.leader, tag=482725763
   [junit4]   2> 7942592 INFO  (coreCloseExecutor-6167-thread-4) [n:127.0.0.1:37860_solr c:hdfsbackuprestore_restored s:shard2 r:core_node62 x:hdfsbackuprestore_restored_shard2_replica_n61] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.hdfsbackuprestore_restored.shard2.leader, tag=718528965
   [junit4]   2> 7942654 INFO  (coreCloseExecutor-6168-thread-4) [n:127.0.0.1:33802_solr c:hdfsbackuprestore s:shard1_1 r:core_node12 x:hdfsbackuprestore_shard1_1_replica_n10] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.hdfsbackuprestore.shard1_1.leader, tag=753244155
   [junit4]   2> 7942655 INFO  (coreCloseExecutor-6168-thread-5) [n:127.0.0.1:33802_solr c:hdfsbackuprestore s:shard1_1 r:core_node14 x:hdfsbackuprestore_shard1_1_replica0] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.hdfsbackuprestore.shard1_1.replica0, tag=613137445
   [junit4]   2> 7942655 INFO  (coreCloseExecutor-6168-thread-5) [n:127.0.0.1:33802_solr c:hdfsbackuprestore s:shard1_1 r:core_node14 x:hdfsbackuprestore_shard1_1_replica0] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1ed191ac: rootName = null, domain = solr.core.hdfsbackuprestore.shard1_1.replica0, service url = null, agent id = null] for registry solr.core.hdfsbackuprestore.shard1_1.replica0 / com.codahale.metrics.MetricRegistry@4470ef53
   [junit4]   2> 7942722 INFO  (coreCloseExecutor-6168-thread-5) [n:127.0.0.1:33802_solr c:hdfsbackuprestore s:shard1_1 r:core_node14 x:hdfsbackuprestore_shard1_1_replica0] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.hdfsbackuprestore.shard1_1.leader, tag=613137445
   [junit4]   2> 7942728 INFO  (coreCloseExecutor-6168-thread-2) [n:127.0.0.1:33802_solr c:hdfsbackuprestore s:shard1 r:core_node2 x:hdfsbackuprestore_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.hdfsbackuprestore.shard1.leader, tag=743151256
   [junit4]   2> 7942763 INFO  (jetty-closer-2378-thread-2) [    ] o.a.s.c.Overseer Overseer (id=98659427661905926-127.0.0.1:37860_solr-n_0000000000) closing
   [junit4]   2> 7942763 INFO  (OverseerStateUpdate-98659427661905926-127.0.0.1:37860_solr-n_0000000000) [n:127.0.0.1:37860_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37860_solr
   [junit4]   2> 7942766 INFO  (zkCallback-2390-thread-3-processing-n:127.0.0.1:37860_solr) [n:127.0.0.1:37860_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 7944357 WARN  (zkCallback-2385-thread-1-processing-n:127.0.0.1:33802_solr) [n:127.0.0.1:33802_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 7944358 INFO  (jetty-closer-2378-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@ee26d8c{/solr,null,UNAVAILABLE}
   [junit4]   2> 7944362 WARN  (zkCallback-2390-thread-3-processing-n:127.0.0.1:37860_solr) [n:127.0.0.1:37860_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 7944363 INFO  (jetty-closer-2378-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5c101e95{/solr,null,UNAVAILABLE}
   [junit4]   2> 7944377 ERROR (SUITE-TestHdfsCloudBackupRestore-seed#[46CBDB43005D3CE8]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 7944947 INFO  (SUITE-TestHdfsCloudBackupRestore-seed#[46CBDB43005D3CE8]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51828 51828
   [junit4]   2> 7945038 INFO  (Thread-3321) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51828 51828
   [junit4]   2> 7945039 WARN  (Thread-3321) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 3 /solr/aliases.json
   [junit4]   2> 2 /solr/security.json
   [junit4]   2> 2 /solr/collections/hdfsbackuprestore_restored/state.json
   [junit4]   2> 2 /solr/configs/conf1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 28 /solr/collections/hdfsbackuprestore/state.json
   [junit4]   2> 3 /solr/clusterstate.json
   [junit4]   2> 3 /solr/clusterprops.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 3 /solr/live_nodes
   [junit4]   2> 3 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.TestHdfsCloudBackupRestore_46CBDB43005D3CE8-001
   [junit4]   2> Sep 14, 2017 9:10:55 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=5, maxDocsPerChunk=5, blockSize=7), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=5, blockSize=7)), sim=RandomSimilarity(queryNorm=true): {}, locale=bg-BG, timezone=America/Antigua
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=2,free=170725416,total=504889344
   [junit4]   2> NOTE: All tests run in this JVM: [TestUseDocValuesAsStored, TestSolrIndexConfig, TemplateUpdateProcessorTest, TestObjectReleaseTracker, SimpleMLTQParserTest, TestCursorMarkWithoutUniqueKey, TestCoreContainer, ConjunctionSolrSpellCheckerTest, SampleTest, IndexSchemaTest, TestComplexPhraseQParserPlugin, TestHashPartitioner, TestRawTransformer, AlternateDirectoryTest, DistributedMLTComponentTest, DistributedFacetExistsSmallTest, SliceStateTest, LeaderInitiatedRecoveryOnShardRestartTest, DirectSolrSpellCheckerTest, TestSweetSpotSimilarityFactory, TestHttpShardHandlerFactory, ShardRoutingTest, RequestLoggingTest, TestStandardQParsers, BasicFunctionalityTest, TestCoreDiscovery, ChaosMonkeySafeLeaderTest, DOMUtilTest, TestDynamicLoading, TestSerializedLuceneMatchVersion, SOLR749Test, HdfsThreadLeakTest, SoftAutoCommitTest, JsonLoaderTest, SolrJmxReporterCloudTest, TestConfigReload, TestShardHandlerFactory, LeaderFailoverAfterPartitionTest, NoCacheHeaderTest, TestFieldCacheWithThreads, OverriddenZkACLAndCredentialsProvidersTest, ZkControllerTest, AnalyticsQueryTest, RuleEngineTest, PrimUtilsTest, TestHighFrequencyDictionaryFactory, TestLegacyNumericUtils, DistributedFacetPivotSmallAdvancedTest, HighlighterTest, TestDownShardTolerantSearch, DistribCursorPagingTest, TestLocalFSCloudBackupRestore, TestCharFilters, TestManagedResource, TestOverriddenPrefixQueryForCustomFieldType, SolrCloudReportersTest, MergeStrategyTest, TestCloudSchemaless, ReplaceNodeTest, TestFiltering, BaseCdcrDistributedZkTest, CoreAdminHandlerTest, DistributedTermsComponentTest, TestExactStatsCache, TestCryptoKeys, ConnectionManagerTest, TestConfigSetsAPI, CdcrReplicationHandlerTest, SolrXmlInZkTest, DistributedIntervalFacetingTest, TestShortCircuitedRequests, DeleteInactiveReplicaTest, OpenCloseCoreStressTest, TestPerFieldSimilarity, FileBasedSpellCheckerTest, CollectionStateFormat2Test, TestPostingsSolrHighlighter, SpatialFilterTest, TestSubQueryTransformer, TestRequestStatusCollectionAPI, LoggingHandlerTest, TestSchemaResource, ShardRoutingCustomTest, DisMaxRequestHandlerTest, TestFieldCacheSort, ConvertedLegacyTest, DistributedQueryComponentCustomSortTest, TestManagedSynonymFilterFactory, TestTestInjection, TestReqParamsAPI, ExplicitHLLTest, ResourceLoaderTest, TestExportWriter, ShardSplitTest, TestFilteredDocIdSet, TestAtomicUpdateErrorCases, TestQueryWrapperFilter, HdfsCollectionsAPIDistributedZkTest, CollectionsAPISolrJTest, TestStressVersions, TestDefaultStatsCache, TestJavabinTupleStreamParser, SuggesterTSTTest, TestCloudPseudoReturnFields, SmileWriterTest, TestConfigSets, DistributedQueueTest, SchemaVersionSpecificBehaviorTest, TestNonDefinedSimilarityFactory, TestDistributedGrouping, TestBulkSchemaAPI, CSVRequestHandlerTest, TestSystemCollAutoCreate, TestSolrCloudWithKerberosAlt, HdfsWriteToMultipleCollectionsTest, SecurityConfHandlerTest, TestElisionMultitermQuery, ShufflingReplicaListTransformerTest, ClusterStateUpdateTest, TestRandomDVFaceting, TestSolr4Spatial, StatsComponentTest, SpellCheckComponentTest, TestGroupingSearch, SolrCmdDistributorTest, TestSort, TestFunctionQuery, TestIndexSearcher, ShowFileRequestHandlerTest, SolrIndexSplitterTest, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SuggesterFSTTest, DocValuesTest, StatelessScriptUpdateProcessorFactoryTest, TestSolrDeletionPolicy1, CacheHeaderTest, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, TestQueryTypes, TestValueSourceCache, TermVectorComponentTest, MoreLikeThisHandlerTest, FastVectorHighlighterTest, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, TestCollationField, ReturnFieldsTest, QueryParsingTest, TestPartialUpdateDeduplication, TestLFUCache, CloudExitableDirectoryReaderTest, ConcurrentDeleteAndCreateCollectionTest, ConfigSetsAPITest, CreateCollectionCleanupTest, DeleteReplicaTest, HttpPartitionTest, MoveReplicaHDFSUlogDirTest, PeerSyncReplicationTest, SaslZkACLProviderTest, TestCloudRecovery, TestHdfsCloudBackupRestore]
   [junit4] Completed [527/731 (8!)] on J1 in 75.38s, 1 test, 1 error <<< FAILURES!

[...truncated 46935 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]