[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 338 - Unstable

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 338 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/338/

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSegmentSorting

Error Message:
Error uploading file /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-update-processor-chains.xml to zookeeper path /configs/class org.apache.solr.cloud.TestSegmentSorting_configSet/solrconfig-update-processor-chains.xml

Stack Trace:
java.io.IOException: Error uploading file /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-update-processor-chains.xml to zookeeper path /configs/class org.apache.solr.cloud.TestSegmentSorting_configSet/solrconfig-update-processor-chains.xml
        at __randomizedtesting.SeedInfo.seed([F2ADC4C129173DEC]:0)
        at org.apache.solr.common.cloud.ZkMaintenanceUtils$1.visitFile(ZkMaintenanceUtils.java:309)
        at org.apache.solr.common.cloud.ZkMaintenanceUtils$1.visitFile(ZkMaintenanceUtils.java:291)
        at java.nio.file.Files.walkFileTree(Files.java:2670)
        at java.nio.file.Files.walkFileTree(Files.java:2742)
        at org.apache.solr.common.cloud.ZkMaintenanceUtils.uploadToZK(ZkMaintenanceUtils.java:291)
        at org.apache.solr.common.cloud.SolrZkClient.uploadToZK(SolrZkClient.java:826)
        at org.apache.solr.common.cloud.ZkConfigManager.uploadConfigDir(ZkConfigManager.java:65)
        at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.uploadConfig(ZkClientClusterStateProvider.java:145)
        at org.apache.solr.cloud.SolrCloudTestCase$Builder.build(SolrCloudTestCase.java:203)
        at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:192)
        at org.apache.solr.cloud.TestSegmentSorting.setupCluster(TestSegmentSorting.java:57)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /configs
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1105)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$2(SolrZkClient.java:305)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:305)
        at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:516)
        at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:424)
        at org.apache.solr.common.cloud.ZkMaintenanceUtils$1.visitFile(ZkMaintenanceUtils.java:305)
        ... 34 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSegmentSorting

Error Message:
ObjectTracker found 33 object(s) that were not released!!! [InternalHttpClient, ZkController, InternalHttpClient, SolrZkClient, SolrZkClient, InternalHttpClient, InternalHttpClient, SolrZkClient, InternalHttpClient, Overseer, InternalHttpClient, SolrZkClient, ZkController, ZkController, InternalHttpClient, InternalHttpClient, SolrZkClient, InternalHttpClient, SolrZkClient, InternalHttpClient, InternalHttpClient, SolrZkClient, InternalHttpClient, SolrZkClient, SolrZkClient, SolrZkClient, InternalHttpClient, InternalHttpClient, ZkController, InternalHttpClient, SolrZkClient, InternalHttpClient, ZkController] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:215)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:539)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:454)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:215)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:539)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:121)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:111)  at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:300)  at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:155)  at org.apache.solr.client.solrj.impl.CloudSolrClient.getZkStateReader(CloudSolrClient.java:342)  at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69)  at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:676)  at org.apache.solr.core.CoreContainer.createMetricsHistoryHandler(CoreContainer.java:755)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:577)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:327)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:545)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:545)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:121)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:111)  at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:300)  at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:155)  at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.uploadConfig(ZkClientClusterStateProvider.java:144)  at org.apache.solr.cloud.SolrCloudTestCase$Builder.build(SolrCloudTestCase.java:203)  at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:192)  at org.apache.solr.cloud.TestSegmentSorting.setupCluster(TestSegmentSorting.java:57)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.lang.reflect.Method.invoke(Method.java:498)  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:545)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.Overseer  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.Overseer.start(Overseer.java:558)  at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:851)  at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)  at org.apache.solr.cloud.ZkController$1.command(ZkController.java:361)  at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:167)  at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)  at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:141)  at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$0(SolrZkClient.java:268)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)  at org.apache.solr.client.solrj.impl.CloudSolrClient.<init>(CloudSolrClient.java:280)  at org.apache.solr.client.solrj.impl.CloudSolrClient$Builder.build(CloudSolrClient.java:1582)  at org.apache.solr.cloud.MiniSolrCloudCluster.buildSolrClient(MiniSolrCloudCluster.java:512)  at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:265)  at org.apache.solr.cloud.SolrCloudTestCase$Builder.build(SolrCloudTestCase.java:200)  at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:192)  at org.apache.solr.cloud.TestSegmentSorting.setupCluster(TestSegmentSorting.java:57)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.lang.reflect.Method.invoke(Method.java:498)  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:121)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:111)  at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:300)  at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:155)  at org.apache.solr.client.solrj.impl.CloudSolrClient.getZkStateReader(CloudSolrClient.java:342)  at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69)  at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:676)  at org.apache.solr.cloud.Overseer.getSolrCloudManager(Overseer.java:575)  at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.<init>(OverseerCollectionMessageHandler.java:205)  at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.getOverseerMessageHandlerSelector(OverseerCollectionConfigSetProcessor.java:87)  at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.<init>(OverseerCollectionConfigSetProcessor.java:70)  at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.<init>(OverseerCollectionConfigSetProcessor.java:41)  at org.apache.solr.cloud.Overseer.start(Overseer.java:545)  at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:851)  at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)  at org.apache.solr.cloud.ZkController.init(ZkController.java:849)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:452)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:454)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:454)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:215)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:539)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:545)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:121)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:111)  at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:300)  at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:155)  at org.apache.solr.client.solrj.impl.CloudSolrClient.getZkStateReader(CloudSolrClient.java:342)  at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69)  at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:676)  at org.apache.solr.core.CoreContainer.createMetricsHistoryHandler(CoreContainer.java:755)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:577)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:545)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:327)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:545)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:545)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:121)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:111)  at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:300)  at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:155)  at org.apache.solr.client.solrj.impl.CloudSolrClient.getZkStateReader(CloudSolrClient.java:342)  at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69)  at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:676)  at org.apache.solr.core.CoreContainer.createMetricsHistoryHandler(CoreContainer.java:755)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:577)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:545)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:327)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:327)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:327)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:545)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:215)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:539)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:454)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:545)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:186)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:121)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:111)  at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:300)  at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:155)  at org.apache.solr.client.solrj.impl.CloudSolrClient.getZkStateReader(CloudSolrClient.java:342)  at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69)  at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:676)  at org.apache.solr.core.CoreContainer.createMetricsHistoryHandler(CoreContainer.java:755)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:577)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:215)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:539)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkController  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:454)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:113)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:555)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 33 object(s) that were not released!!! [InternalHttpClient, ZkController, InternalHttpClient, SolrZkClient, SolrZkClient, InternalHttpClient, InternalHttpClient, SolrZkClient, InternalHttpClient, Overseer, InternalHttpClient, SolrZkClient, ZkController, ZkController, InternalHttpClient, InternalHttpClient, SolrZkClient, InternalHttpClient, SolrZkClient, InternalHttpClient, InternalHttpClient, SolrZkClient, InternalHttpClient, SolrZkClient, SolrZkClient, SolrZkClient, InternalHttpClient, InternalHttpClient, ZkController, InternalHttpClient, SolrZkClient, InternalHttpClient, ZkController]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
        at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:215)
        at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:539)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:252)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:172)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:741)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1477)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1542)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1186)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1023)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:473)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:308)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:396)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:369)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:392)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:248)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.ap

[...truncated too long message...]

xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null2087955340
     [copy] Copying 239 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null2087955340
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null2087955340

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 105ms :: artifacts dl 21ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 467 minutes 18 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 339 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/339/

1 tests failed.
FAILED:  org.apache.solr.update.TestInPlaceUpdatesDistrib.test

Error Message:
Error from server at https://127.0.0.1:45770/_uzf/collection1: ERROR adding document SolrInputDocument(fields: [id=-1333, title_s=title-1333, id_i=-1333, _version_=1613631513010634752])

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:45770/_uzf/collection1: ERROR adding document SolrInputDocument(fields: [id=-1333, title_s=title-1333, id_i=-1333, _version_=1613631513010634752])
        at __randomizedtesting.SeedInfo.seed([9C9120F0BD897DF4:14C51F2A1375100C]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.addDocAndGetVersion(TestInPlaceUpdatesDistrib.java:1107)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.buildRandomIndex(TestInPlaceUpdatesDistrib.java:1152)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:320)
        at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:146)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13999 lines...]
   [junit4] Suite: org.apache.solr.update.TestInPlaceUpdatesDistrib
   [junit4]   2> 1580169 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/init-core-data-001
   [junit4]   2> 1580169 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 1580170 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1580172 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1580173 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_uzf/
   [junit4]   2> 1580174 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.SolrTestCaseJ4 updateLog impl=solr.CdcrUpdateLog
   [junit4]   2> 1580175 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 1580175 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1580206 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1580219 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=inplace-updates
   [junit4]   2> 1580223 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 1580350 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1580350 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1580373 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1580385 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1580385 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1580387 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 1580425 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1580438 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 1580443 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 1580443 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 1580443 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1580443 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/init-core-data-001/]
   [junit4]   2> 1580454 WARN  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1580547 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 1580547 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1580548 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1580548 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1580601 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@11deaace[collection1] main]
   [junit4]   2> 1580602 WARN  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Cannot write to config directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead.
   [junit4]   2> 1580603 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1580603 INFO  (coreLoadExecutor-5336-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613631401135964160
   [junit4]   2> 1580606 INFO  (searcherExecutor-5337-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@11deaace[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1580607 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[9C9120F0BD897DF4]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 1580611 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1580611 INFO  (Thread-3127) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1580611 INFO  (Thread-3127) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1580619 ERROR (Thread-3127) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1580711 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ZkTestServer start zk server on port:41013
   [junit4]   2> 1580720 INFO  (zkConnectionManagerCallback-5237-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1580725 INFO  (zkConnectionManagerCallback-5239-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1580732 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1580734 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema-inplace-updates.xml to /configs/conf1/schema.xml
   [junit4]   2> 1580735 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1580737 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1580739 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1580740 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1580742 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1580743 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1580745 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1580746 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1580747 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1580751 INFO  (zkConnectionManagerCallback-5243-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1580753 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1580931 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1580932 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1580932 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1580932 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1580935 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@428b142d{/_uzf,null,AVAILABLE}
   [junit4]   2> 1580935 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5ce7c5a7{SSL,[ssl, http/1.1]}{127.0.0.1:39925}
   [junit4]   2> 1580935 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.Server Started @1581001ms
   [junit4]   2> 1580935 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/tempDir-001/control/data, replicaType=NRT, hostContext=/_uzf, hostPort=39925, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/control-001/cores}
   [junit4]   2> 1580936 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1580936 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1580936 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1580936 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1580937 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1580937 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-07T02:20:23.744Z
   [junit4]   2> 1580939 INFO  (zkConnectionManagerCallback-5245-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1580940 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1580940 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/control-001/solr.xml
   [junit4]   2> 1580945 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1580945 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1580946 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1581026 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41013/solr
   [junit4]   2> 1581028 INFO  (zkConnectionManagerCallback-5249-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581030 INFO  (zkConnectionManagerCallback-5251-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581137 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39925__uzf
   [junit4]   2> 1581139 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.Overseer Overseer (id=72058486221897733-127.0.0.1:39925__uzf-n_0000000000) starting
   [junit4]   2> 1581149 INFO  (zkConnectionManagerCallback-5258-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581152 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41013/solr ready
   [junit4]   2> 1581153 INFO  (OverseerStateUpdate-72058486221897733-127.0.0.1:39925__uzf-n_0000000000) [n:127.0.0.1:39925__uzf    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:39925__uzf
   [junit4]   2> 1581153 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39925__uzf
   [junit4]   2> 1581154 INFO  (OverseerStateUpdate-72058486221897733-127.0.0.1:39925__uzf-n_0000000000) [n:127.0.0.1:39925__uzf    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1581167 INFO  (zkCallback-5257-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1581182 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1581211 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1581227 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1581227 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1581229 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/control-001/cores
   [junit4]   2> 1581272 INFO  (zkConnectionManagerCallback-5264-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1581273 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1581274 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41013/solr ready
   [junit4]   2> 1581300 INFO  (qtp2106946115-16319) [n:127.0.0.1:39925__uzf    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:39925__uzf&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1581304 INFO  (OverseerThreadFactory-5350-thread-1) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1581424 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1581425 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1582447 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1582461 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=inplace-updates
   [junit4]   2> 1582465 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 1582465 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1582466 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1582466 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1582480 WARN  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1582553 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 1582553 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1582555 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1582555 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1582609 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2d4b3047[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1582610 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1582611 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1582612 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1582612 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613631403242553344
   [junit4]   2> 1582613 INFO  (searcherExecutor-5355-thread-1-processing-n:127.0.0.1:39925__uzf x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@2d4b3047[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1582619 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 1582623 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1582623 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1582623 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:39925/_uzf/control_collection_shard1_replica_n1/
   [junit4]   2> 1582624 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1582624 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:39925/_uzf/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1582624 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1582628 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:39925/_uzf/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1582736 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1582739 INFO  (qtp2106946115-16320) [n:127.0.0.1:39925__uzf c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1314
   [junit4]   2> 1582743 INFO  (qtp2106946115-16319) [n:127.0.0.1:39925__uzf    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1582839 INFO  (zkCallback-5250-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1583308 INFO  (OverseerCollectionConfigSetProcessor-72058486221897733-127.0.0.1:39925__uzf-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1583744 INFO  (qtp2106946115-16319) [n:127.0.0.1:39925__uzf    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:39925__uzf&wt=javabin&version=2} status=0 QTime=2444
   [junit4]   2> 1583749 INFO  (zkConnectionManagerCallback-5269-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1583751 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1583752 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41013/solr ready
   [junit4]   2> 1583752 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1583757 INFO  (qtp2106946115-16317) [n:127.0.0.1:39925__uzf    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1583760 INFO  (OverseerThreadFactory-5350-thread-2) [    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 1583966 WARN  (OverseerThreadFactory-5350-thread-2) [    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 1583968 INFO  (qtp2106946115-16317) [n:127.0.0.1:39925__uzf    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1583969 INFO  (qtp2106946115-16317) [n:127.0.0.1:39925__uzf    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=211
   [junit4]   2> 1584135 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-1-001 of type NRT
   [junit4]   2> 1584137 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1584138 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1584138 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1584138 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1584139 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@63064814{/_uzf,null,AVAILABLE}
   [junit4]   2> 1584140 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@14aa7053{SSL,[ssl, http/1.1]}{127.0.0.1:45770}
   [junit4]   2> 1584140 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.Server Started @1584205ms
   [junit4]   2> 1584140 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/_uzf, hostPort=45770, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-1-001/cores}
   [junit4]   2> 1584140 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1584140 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1584140 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1584140 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1584140 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1584140 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-07T02:20:26.947Z
   [junit4]   2> 1584143 INFO  (zkConnectionManagerCallback-5271-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1584144 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1584144 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-1-001/solr.xml
   [junit4]   2> 1584149 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1584149 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1584150 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1584558 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41013/solr
   [junit4]   2> 1584560 INFO  (zkConnectionManagerCallback-5275-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1584563 INFO  (zkConnectionManagerCallback-5277-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1584571 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1584576 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1584576 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45770__uzf
   [junit4]   2> 1584578 INFO  (zkCallback-5250-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1584578 INFO  (zkCallback-5257-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1584578 INFO  (zkCallback-5268-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1584578 INFO  (zkCallback-5276-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1584605 INFO  (zkConnectionManagerCallback-5284-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1584606 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1584608 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41013/solr ready
   [junit4]   2> 1584608 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1584636 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1584653 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1584653 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1584655 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-1-001/cores
   [junit4]   2> 1584708 INFO  (qtp2106946115-16319) [n:127.0.0.1:39925__uzf    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:45770__uzf&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1584711 INFO  (OverseerCollectionConfigSetProcessor-72058486221897733-127.0.0.1:39925__uzf-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1584712 INFO  (OverseerThreadFactory-5350-thread-3) [ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:45770__uzf for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 1584738 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf    x:collection1_shard1_replica_n21] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1585764 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1585779 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema [collection1_shard1_replica_n21] Schema name=inplace-updates
   [junit4]   2> 1585783 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 1585783 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n21' using configuration from collection collection1, trusted=true
   [junit4]   2> 1585784 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n21' (registry 'solr.core.collection1.shard1.replica_n21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1585784 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrCore [[collection1_shard1_replica_n21] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-1-001/cores/collection1_shard1_replica_n21], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-1-001/cores/collection1_shard1_replica_n21/data/]
   [junit4]   2> 1585795 WARN  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1585870 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 1585870 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1585872 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1585872 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1585925 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.SolrIndexSearcher Opening [Searcher@56983778[collection1_shard1_replica_n21] main]
   [junit4]   2> 1585927 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1585927 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1585928 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1585929 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613631406720679936
   [junit4]   2> 1585930 INFO  (searcherExecutor-5369-thread-1-processing-n:127.0.0.1:45770__uzf x:collection1_shard1_replica_n21 c:collection1 s:shard1) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrCore [collection1_shard1_replica_n21] Registered new searcher Searcher@56983778[collection1_shard1_replica_n21] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1585935 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node22=0}, version=0}
   [junit4]   2> 1585939 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1585939 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1585939 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:45770/_uzf/collection1_shard1_replica_n21/
   [junit4]   2> 1585939 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1585939 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy https://127.0.0.1:45770/_uzf/collection1_shard1_replica_n21/ has no replicas
   [junit4]   2> 1585939 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1585943 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:45770/_uzf/collection1_shard1_replica_n21/ shard1
   [junit4]   2> 1586095 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1586098 INFO  (qtp367804920-16374) [n:127.0.0.1:45770__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1359
   [junit4]   2> 1586115 INFO  (qtp2106946115-16319) [n:127.0.0.1:39925__uzf c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:45770__uzf&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1407
   [junit4]   2> 1586305 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-2-001 of type NRT
   [junit4]   2> 1586310 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1586311 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1586311 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1586311 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1586311 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5dcaebd1{/_uzf,null,AVAILABLE}
   [junit4]   2> 1586312 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@32e6ee71{SSL,[ssl, http/1.1]}{127.0.0.1:34449}
   [junit4]   2> 1586312 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.Server Started @1586378ms
   [junit4]   2> 1586312 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/_uzf, hostPort=34449, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-2-001/cores}
   [junit4]   2> 1586313 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1586319 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1586320 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1586320 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1586320 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1586320 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-07T02:20:29.127Z
   [junit4]   2> 1586330 INFO  (zkConnectionManagerCallback-5287-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1586332 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1586332 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-2-001/solr.xml
   [junit4]   2> 1586337 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1586337 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1586339 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1586574 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41013/solr
   [junit4]   2> 1586576 INFO  (zkConnectionManagerCallback-5291-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1586579 INFO  (zkConnectionManagerCallback-5293-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1586596 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1586602 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1586602 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34449__uzf
   [junit4]   2> 1586603 INFO  (zkCallback-5257-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1586603 INFO  (zkCallback-5268-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1586603 INFO  (zkCallback-5283-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1586603 INFO  (zkCallback-5276-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1586603 INFO  (zkCallback-5250-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1586616 INFO  (zkCallback-5292-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1586653 INFO  (zkConnectionManagerCallback-5300-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1586654 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1586655 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41013/solr ready
   [junit4]   2> 1586656 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1586683 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1586700 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1586700 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1586703 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-2-001/cores
   [junit4]   2> 1586714 INFO  (OverseerCollectionConfigSetProcessor-72058486221897733-127.0.0.1:39925__uzf-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1586793 INFO  (qtp1750885238-16411) [n:127.0.0.1:34449__uzf    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34449__uzf&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1586796 INFO  (OverseerThreadFactory-5350-thread-4) [ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:34449__uzf for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 1586808 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf    x:collection1_shard1_replica_n23] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n23&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1587827 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1587841 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.s.IndexSchema [collection1_shard1_replica_n23] Schema name=inplace-updates
   [junit4]   2> 1587845 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 1587845 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n23' using configuration from collection collection1, trusted=true
   [junit4]   2> 1587846 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n23' (registry 'solr.core.collection1.shard1.replica_n23') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1587846 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.c.SolrCore [[collection1_shard1_replica_n23] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-2-001/cores/collection1_shard1_replica_n23], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-2-001/cores/collection1_shard1_replica_n23/data/]
   [junit4]   2> 1587857 WARN  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1587927 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 1587927 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1587928 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1587929 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1587983 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.s.SolrIndexSearcher Opening [Searcher@5229a870[collection1_shard1_replica_n23] main]
   [junit4]   2> 1587984 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1587985 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1587987 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1587987 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613631408878649344
   [junit4]   2> 1587989 INFO  (searcherExecutor-5383-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.c.SolrCore [collection1_shard1_replica_n23] Registered new searcher Searcher@5229a870[collection1_shard1_replica_n23] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1587994 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node24=0, core_node22=0}, version=1}
   [junit4]   2> 1587996 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.c.ZkController Core needs to recover:collection1_shard1_replica_n23
   [junit4]   2> 1587996 INFO  (updateExecutor-5288-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 1587996 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 1587997 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy startupVersions is empty
   [junit4]   2> 1587999 INFO  (qtp1750885238-16412) [n:127.0.0.1:34449__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n23] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n23&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1190
   [junit4]   2> 1588008 INFO  (qtp1750885238-16411) [n:127.0.0.1:34449__uzf c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:34449__uzf&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1214
   [junit4]   2> 1588008 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf c:collection1 s:shard1 r:core_node22 x:collection1_shard1_replica_n21] o.a.s.c.S.Request [collection1_shard1_replica_n21]  webapp=/_uzf path=/admin/ping params={wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1588008 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf c:collection1 s:shard1 r:core_node22 x:collection1_shard1_replica_n21] o.a.s.c.S.Request [collection1_shard1_replica_n21]  webapp=/_uzf path=/admin/ping params={wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1588008 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1_shard1_replica_n23]
   [junit4]   2> 1588009 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 1588009 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Publishing state of core [collection1_shard1_replica_n23] as recovering, leader is [https://127.0.0.1:45770/_uzf/collection1_shard1_replica_n21/] and I am [https://127.0.0.1:34449/_uzf/collection1_shard1_replica_n23/]
   [junit4]   2> 1588010 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node24_recovering=0, core_node24=0, core_node22=0}, version=2}
   [junit4]   2> 1588011 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:45770/_uzf]; [WaitForState: action=PREPRECOVERY&core=collection1_shard1_replica_n21&nodeName=127.0.0.1:34449__uzf&coreNodeName=core_node24&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 1588020 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf    x:collection1_shard1_replica_n21] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node24, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true, maxTime: 183 s
   [junit4]   2> 1588021 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf    x:collection1_shard1_replica_n21] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1_shard1_replica_n21, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:34449__uzf, coreNodeName=core_node24, onlyIfActiveCheckResult=false, nodeProps: core_node24:{"core":"collection1_shard1_replica_n23","base_url":"https://127.0.0.1:34449/_uzf","node_name":"127.0.0.1:34449__uzf","state":"down","type":"NRT"}
   [junit4]   2> 1588183 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-3-001 of type NRT
   [junit4]   2> 1588184 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1588186 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1588186 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1588186 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1588186 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7926cd60{/_uzf,null,AVAILABLE}
   [junit4]   2> 1588187 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5fd24699{SSL,[ssl, http/1.1]}{127.0.0.1:34962}
   [junit4]   2> 1588187 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.e.j.s.Server Started @1588253ms
   [junit4]   2> 1588187 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/_uzf, hostPort=34962, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-3-001/cores}
   [junit4]   2> 1588187 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1588188 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1588188 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1588188 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1588188 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1588188 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-07T02:20:30.995Z
   [junit4]   2> 1588190 INFO  (zkConnectionManagerCallback-5304-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1588192 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1588192 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-3-001/solr.xml
   [junit4]   2> 1588196 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1588197 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1588198 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1588272 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41013/solr
   [junit4]   2> 1588274 INFO  (zkConnectionManagerCallback-5308-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1588277 INFO  (zkConnectionManagerCallback-5310-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1588286 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1588291 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1588291 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34962__uzf
   [junit4]   2> 1588293 INFO  (zkCallback-5257-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1588293 INFO  (zkCallback-5299-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1588293 INFO  (zkCallback-5292-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1588293 INFO  (zkCallback-5276-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1588293 INFO  (zkCallback-5268-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1588293 INFO  (zkCallback-5250-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1588293 INFO  (zkCallback-5283-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1588293 INFO  (zkCallback-5309-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1588323 INFO  (zkConnectionManagerCallback-5317-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1588324 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1588325 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41013/solr ready
   [junit4]   2> 1588326 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1588357 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1588375 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1588375 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fde87c2
   [junit4]   2> 1588378 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[9C9120F0BD897DF4]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9C9120F0BD897DF4-001/shard-3-001/cores
   [junit4]   2> 1588468 INFO  (qtp1750885238-16409) [n:127.0.0.1:34449__uzf    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34962__uzf&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1588471 INFO  (OverseerCollectionConfigSetProcessor-72058486221897733-127.0.0.1:39925__uzf-n_0000000000) [    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1588472 INFO  (OverseerThreadFactory-5350-thread-5) [ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:34962__uzf for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 1588497 INFO  (qtp371390707-16453) [n:127.0.0.1:34962__uzf    x:collection1_shard1_replica_n25] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n25&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1589021 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf    x:collection1_shard1_replica_n21] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1_shard1_replica_n21, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:34449__uzf, coreNodeName=core_node24, onlyIfActiveCheckResult=false, nodeProps: core_node24:{"core":"collection1_shard1_replica_n23","base_url":"https://127.0.0.1:34449/_uzf","node_name":"127.0.0.1:34449__uzf","state":"recovering","type":"NRT"}
   [junit4]   2> 1589021 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf    x:collection1_shard1_replica_n21] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node24, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 1589021 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf    x:collection1_shard1_replica_n21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:34449__uzf&onlyIfLeaderActive=true&core=collection1_shard1_replica_n21&coreNodeName=core_node24&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
   [junit4]   2> 1589518 INFO  (qtp371390707-16453) [n:127.0.0.1:34962__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n25] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1589523 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:45770/_uzf/collection1_shard1_replica_n21/] - recoveringAfterStartup=[true]
   [junit4]   2> 1589523 WARN  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.u.PeerSyncWithLeader no frame of reference to tell if we've missed updates
   [junit4]   2> 1589523 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy PeerSync Recovery was not successful - trying replication.
   [junit4]   2> 1589523 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Starting Replication Recovery.
   [junit4]   2> 1589523 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Attempting to replicate from [https://127.0.0.1:45770/_uzf/collection1_shard1_replica_n21/].
   [junit4]   2> 1589532 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf c:collection1 s:shard1 r:core_node22 x:collection1_shard1_replica_n21] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1613631410498699264,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1589533 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf c:collection1 s:shard1 r:core_node22 x:collection1_shard1_replica_n21] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 1589535 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf c:collection1 s:shard1 r:core_node22 x:collection1_shard1_replica_n21] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1589535 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf c:collection1 s:shard1 r:core_node22 x:collection1_shard1_replica_n21] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n21]  webapp=/_uzf path=/update params={waitSearcher=true&openSearcher=false&commit=true&softCommit=false&commit_end_point=true&wt=javabin&version=2}{commit=} 0 3
   [junit4]   2> 1589537 INFO  (qtp371390707-16453) [n:127.0.0.1:34962__uzf c:collection1 s:shard1  x:collection1_shard1_replica_n25] o.a.s.s.IndexSchema [collection1_shard1_replica_n25] Schema name=inplace-updates
   [junit4]   2> 1589540 INFO  (qtp367804920-16375) [n:127.0.0.1:45770__uzf c:collection1 s:shard1 r:core_node22 x:collection1_shard1_replica_n21] o.a.s.c.S.Request [collection1_shard1_replica_n21]  webapp=/_uzf path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1589541 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 1589541 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 1589541 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:127.0.0.1:34449__uzf c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1589541 INFO  (recoveryExecutor-5289-thread-1-processing-n:127.0.0.1:34449__uzf x:collection1_shard1_replica_n23 c:collection1 s:shard1 r:core_node24) [n:12

[...truncated too long message...]

checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null183231716
     [copy] Copying 239 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null183231716
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null183231716

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 45ms :: artifacts dl 6ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 278 minutes 0 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 340 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/340/

2 tests failed.
FAILED:  org.apache.solr.cloud.AddReplicaTest.testAddMultipleReplicas

Error Message:
Task with the same requestid already exists.

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Task with the same requestid already exists.
        at __randomizedtesting.SeedInfo.seed([2F861220881AA566:C8D6CDFA38E99D08]:0)
        at org.apache.solr.client.solrj.request.CollectionAdminRequest$AsyncCollectionAdminRequest.processAsync(CollectionAdminRequest.java:199)
        at org.apache.solr.client.solrj.request.CollectionAdminRequest$AsyncCollectionAdminRequest.processAndWait(CollectionAdminRequest.java:226)
        at org.apache.solr.cloud.AddReplicaTest.testAddMultipleReplicas(AddReplicaTest.java:88)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost

Error Message:
last ClusterState: znodeVersion: 235 live nodes:[127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10036_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10052_solr, 127.0.0.1:10006_solr, 127.0.0.1:10079_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10098_solr, 127.0.0.1:10000_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10003_solr, 127.0.0.1:10011_solr, 127.0.0.1:10001_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10071_solr, 127.0.0.1:10063_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10015_solr, 127.0.0.1:10055_solr, 127.0.0.1:10069_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10062_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10002_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10037_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr] collections:{.system=DocCollection(.system//clusterstate.json/234)={   "replicationFactor":"3",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{"shard1":{       "replicas":{         "core_node1":{           "core":".system_shard1_replica_n1",           "SEARCHER.searcher.maxDoc":139,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":79740,           "node_name":"127.0.0.1:10022_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":7.426366209983826E-5,           "SEARCHER.searcher.numDocs":139},         "core_node2":{           "core":".system_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":139,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":79740,           "node_name":"127.0.0.1:10047_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":7.426366209983826E-5,           "SEARCHER.searcher.numDocs":139},         "core_node3":{           "core":".system_shard1_replica_n3",           "leader":"true",           "SEARCHER.searcher.maxDoc":139,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":79740,           "node_name":"127.0.0.1:10097_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":7.426366209983826E-5,           "SEARCHER.searcher.numDocs":139}},       "range":"80000000-7fffffff",       "state":"active"}}}, testNodeLost=DocCollection(testNodeLost//clusterstate.json/234)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"5",   "autoAddReplicas":"false",   "nrtReplicas":"10",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard20":{       "replicas":{         "core_node191":{           "core":"testNodeLost_shard20_replica_n191",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node200":{           "core":"testNodeLost_shard20_replica_n200",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10027_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node199":{           "core":"testNodeLost_shard20_replica_n199",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10058_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node198":{           "core":"testNodeLost_shard20_replica_n198",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10076_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node197":{           "core":"testNodeLost_shard20_replica_n197",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10028_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node196":{           "core":"testNodeLost_shard20_replica_n196",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10035_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node195":{           "core":"testNodeLost_shard20_replica_n195",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node194":{           "core":"testNodeLost_shard20_replica_n194",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10040_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node193":{           "core":"testNodeLost_shard20_replica_n193",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10087_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node192":{           "core":"testNodeLost_shard20_replica_n192",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10038_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"73330000-7fffffff",       "state":"active"},     "shard10":{       "replicas":{         "core_node92":{           "core":"testNodeLost_shard10_replica_n92",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10038_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node91":{           "core":"testNodeLost_shard10_replica_n91",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node94":{           "core":"testNodeLost_shard10_replica_n94",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10040_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node93":{           "core":"testNodeLost_shard10_replica_n93",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10087_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node96":{           "core":"testNodeLost_shard10_replica_n96",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10035_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node95":{           "core":"testNodeLost_shard10_replica_n95",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10001_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node98":{           "core":"testNodeLost_shard10_replica_n98",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10076_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node100":{           "core":"testNodeLost_shard10_replica_n100",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10027_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node97":{           "core":"testNodeLost_shard10_replica_n97",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10028_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node99":{           "core":"testNodeLost_shard10_replica_n99",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10058_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"f3330000-ffffffff",       "state":"active"},     "shard11":{       "replicas":{         "core_node108":{           "core":"testNodeLost_shard11_replica_n108",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10085_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node107":{           "core":"testNodeLost_shard11_replica_n107",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10074_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node106":{           "core":"testNodeLost_shard11_replica_n106",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10019_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node105":{           "core":"testNodeLost_shard11_replica_n105",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10075_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node104":{           "core":"testNodeLost_shard11_replica_n104",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10098_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node103":{           "core":"testNodeLost_shard11_replica_n103",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10064_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node102":{           "core":"testNodeLost_shard11_replica_n102",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10069_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node101":{           "core":"testNodeLost_shard11_replica_n101",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10077_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node110":{           "core":"testNodeLost_shard11_replica_n110",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10042_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node109":{           "core":"testNodeLost_shard11_replica_n109",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10056_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"0-ccbffff",       "state":"active"},     "shard12":{       "replicas":{         "core_node119":{           "core":"testNodeLost_shard12_replica_n119",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10062_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node118":{           "core":"testNodeLost_shard12_replica_n118",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10057_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node117":{           "core":"testNodeLost_shard12_replica_n117",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10007_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node116":{           "core":"testNodeLost_shard12_replica_n116",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10010_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node115":{           "core":"testNodeLost_shard12_replica_n115",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10011_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node114":{           "core":"testNodeLost_shard12_replica_n114",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node113":{           "core":"testNodeLost_shard12_replica_n113",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10016_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node112":{           "core":"testNodeLost_shard12_replica_n112",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10022_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node111":{           "core":"testNodeLost_shard12_replica_n111",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10033_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node120":{           "core":"testNodeLost_shard12_replica_n120",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10030_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"ccc0000-1998ffff",       "state":"active"},     "shard13":{       "replicas":{         "core_node129":{           "core":"testNodeLost_shard13_replica_n129",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10080_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node128":{           "core":"testNodeLost_shard13_replica_n128",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10081_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node127":{           "core":"testNodeLost_shard13_replica_n127",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10083_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node126":{           "core":"testNodeLost_shard13_replica_n126",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10026_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node125":{           "core":"testNodeLost_shard13_replica_n125",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10078_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node124":{           "core":"testNodeLost_shard13_replica_n124",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10099_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node123":{           "core":"testNodeLost_shard13_replica_n123",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10046_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node122":{           "core":"testNodeLost_shard13_replica_n122",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10029_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node121":{           "core":"testNodeLost_shard13_replica_n121",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10017_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node130":{           "core":"testNodeLost_shard13_replica_n130",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10044_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"19990000-2665ffff",       "state":"active"},     "shard2":{       "replicas":{         "core_node18":{           "core":"testNodeLost_shard2_replica_n18",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10057_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node17":{           "core":"testNodeLost_shard2_replica_n17",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10007_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node19":{           "core":"testNodeLost_shard2_replica_n19",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10062_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node20":{           "core":"testNodeLost_shard2_replica_n20",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10030_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node12":{           "core":"testNodeLost_shard2_replica_n12",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10022_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node11":{           "core":"testNodeLost_shard2_replica_n11",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10033_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node14":{           "core":"testNodeLost_shard2_replica_n14",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10002_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node13":{           "core":"testNodeLost_shard2_replica_n13",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10016_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node16":{           "core":"testNodeLost_shard2_replica_n16",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10010_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node15":{           "core":"testNodeLost_shard2_replica_n15",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10011_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"8ccc0000-9998ffff",       "state":"active"},     "shard14":{       "replicas":{         "core_node139":{           "core":"testNodeLost_shard14_replica_n139",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10061_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node138":{           "core":"testNodeLost_shard14_replica_n138",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10008_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node137":{           "core":"testNodeLost_shard14_replica_n137",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10095_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node136":{           "core":"testNodeLost_shard14_replica_n136",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node135":{           "core":"testNodeLost_shard14_replica_n135",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10045_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node134":{           "core":"testNodeLost_shard14_replica_n134",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10036_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node133":{           "core":"testNodeLost_shard14_replica_n133",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10023_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node132":{           "core":"testNodeLost_shard14_replica_n132",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10050_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node131":{           "core":"testNodeLost_shard14_replica_n131",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10024_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node140":{           "core":"testNodeLost_shard14_replica_n140",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10086_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"26660000-3332ffff",       "state":"active"},     "shard3":{       "replicas":{         "core_node29":{           "core":"testNodeLost_shard3_replica_n29",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10080_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node28":{           "core":"testNodeLost_shard3_replica_n28",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10081_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node30":{           "core":"testNodeLost_shard3_replica_n30",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10044_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node21":{           "core":"testNodeLost_shard3_replica_n21",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10017_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node23":{           "core":"testNodeLost_shard3_replica_n23",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10046_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node22":{           "core":"testNodeLost_shard3_replica_n22",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10029_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node25":{           "core":"testNodeLost_shard3_replica_n25",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10078_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node24":{           "core":"testNodeLost_shard3_replica_n24",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10099_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node27":{           "core":"testNodeLost_shard3_replica_n27",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10083_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node26":{           "core":"testNodeLost_shard3_replica_n26",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10026_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"99990000-a665ffff",       "state":"active"},     "shard4":{       "replicas":{         "core_node39":{           "core":"testNodeLost_shard4_replica_n39",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10061_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node40":{           "core":"testNodeLost_shard4_replica_n40",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10086_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node32":{           "core":"testNodeLost_shard4_replica_n32",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10050_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node31":{           "core":"testNodeLost_shard4_replica_n31",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10024_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node34":{           "core":"testNodeLost_shard4_replica_n34",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10036_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node33":{           "core":"testNodeLost_shard4_replica_n33",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10023_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node36":{           "core":"testNodeLost_shard4_replica_n36",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10003_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node35":{           "core":"testNodeLost_shard4_replica_n35",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10045_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node38":{           "core":"testNodeLost_shard4_replica_n38",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10008_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node37":{           "core":"testNodeLost_shard4_replica_n37",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10095_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"a6660000-b332ffff",       "state":"active"},     "shard15":{       "replicas":{         "core_node149":{           "core":"testNodeLost_shard15_replica_n149",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10054_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node148":{           "core":"testNodeLost_shard15_replica_n148",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10020_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node147":{           "core":"testNodeLost_shard15_replica_n147",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10013_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node146":{           "core":"testNodeLost_shard15_replica_n146",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10043_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node145":{           "core":"testNodeLost_shard15_replica_n145",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10049_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node144":{           "core":"testNodeLost_shard15_replica_n144",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10032_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node143":{           "core":"testNodeLost_shard15_replica_n143",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10005_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node142":{           "core":"testNodeLost_shard15_replica_n142",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10079_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node141":{           "core":"testNodeLost_shard15_replica_n141",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10088_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node150":{           "core":"testNodeLost_shard15_replica_n150",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10048_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"33330000-3fffffff",       "state":"active"},     "shard16":{       "replicas":{         "core_node159":{           "core":"testNodeLost_shard16_replica_n159",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10015_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node158":{           "core":"testNodeLost_shard16_replica_n158",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10059_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node157":{           "core":"testNodeLost_shard16_replica_n157",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10091_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node156":{           "core":"testNodeLost_shard16_replica_n156",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10031_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node155":{           "core":"testNodeLost_shard16_replica_n155",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10055_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node154":{           "core":"testNodeLost_shard16_replica_n154",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10037_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node153":{           "core":"testNodeLost_shard16_replica_n153",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10060_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node152":{           "core":"testNodeLost_shard16_replica_n152",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10021_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node151":{           "core":"testNodeLost_shard16_replica_n151",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10096_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node160":{           "core":"testNodeLost_shard16_replica_n160",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10073_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"40000000-4ccbffff",       "state":"active"},     "shard5":{       "replicas":{         "core_node50":{           "core":"testNodeLost_shard5_replica_n50",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10048_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node41":{           "core":"testNodeLost_shard5_replica_n41",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10088_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node43":{           "core":"testNodeLost_shard5_replica_n43",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10005_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node42":{           "core":"testNodeLost_shard5_replica_n42",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10079_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node45":{           "core":"testNodeLost_shard5_replica_n45",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10049_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node44":{           "core":"testNodeLost_shard5_replica_n44",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10032_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node47":{           "core":"testNodeLost_shard5_replica_n47",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10013_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node46":{           "core":"testNodeLost_shard5_replica_n46",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10043_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node49":{           "core":"testNodeLost_shard5_replica_n49",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10054_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node48":{           "core":"testNodeLost_shard5_replica_n48",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10020_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"b3330000-bfffffff",       "state":"active"},     "shard17":{       "replicas":{         "core_node169":{           "core":"testNodeLost_shard17_replica_n169",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10006_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node168":{           "core":"testNodeLost_shard17_replica_n168",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10039_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node167":{           "core":"testNodeLost_shard17_replica_n167",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10093_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node166":{           "core":"testNodeLost_shard17_replica_n166",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10009_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node165":{           "core":"testNodeLost_shard17_replica_n165",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10018_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node164":{           "core":"testNodeLost_shard17_replica_n164",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10065_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node163":{           "core":"testNodeLost_shard17_replica_n163",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10068_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node162":{           "core":"testNodeLost_shard17_replica_n162",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10067_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node161":{           "core":"testNodeLost_shard17_replica_n161",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10089_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node170":{           "core":"testNodeLost_shard17_replica_n170",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10012_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"4ccc0000-5998ffff",       "state":"active"},     "shard18":{       "replicas":{         "core_node180":{           "core":"testNodeLost_shard18_replica_n180",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10047_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node179":{           "core":"testNodeLost_shard18_replica_n179",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10090_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node178":{           "core":"testNodeLost_shard18_replica_n178",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10092_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node177":{           "core":"testNodeLost_shard18_replica_n177",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10052_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node176":{           "core":"testNodeLost_shard18_replica_n176",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10063_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node175":{           "core":"testNodeLost_shard18_replica_n175",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10070_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node174":{           "core":"testNodeLost_shard18_replica_n174",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10041_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node173":{           "core":"testNodeLost_shard18_replica_n173",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10004_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node172":{           "core":"testNodeLost_shard18_replica_n172",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10094_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node171":{           "core":"testNodeLost_shard18_replica_n171",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10025_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"59990000-6665ffff",       "state":"active"},     "shard19":{       "replicas":{         "core_node190":{           "core":"testNodeLost_shard19_replica_n190",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10084_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node202":{           "node_name":"127.0.0.1:10072_solr",           "core":"testNodeLost_shard19_replica_n201",           "state":"active",           "INDEX.sizeInBytes":10240,           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6},         "core_node188":{           "core":"testNodeLost_shard19_replica_n188",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10097_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node187":{           "core":"testNodeLost_shard19_replica_n187",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10014_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node186":{           "core":"testNodeLost_shard19_replica_n186",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10071_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node185":{           "core":"testNodeLost_shard19_replica_n185",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10053_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node184":{           "core":"testNodeLost_shard19_replica_n184",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10034_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node183":{           "core":"testNodeLost_shard19_replica_n183",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10066_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node182":{           "core":"testNodeLost_shard19_replica_n182",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10051_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node181":{           "core":"testNodeLost_shard19_replica_n181",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10072_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"66660000-7332ffff",       "state":"active"},     "shard1":{       "replicas":{         "core_node1":{           "core":"testNodeLost_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10077_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node6":{           "core":"testNodeLost_shard1_replica_n6",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10019_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node10":{           "core":"testNodeLost_shard1_replica_n10",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10042_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node7":{           "core":"testNodeLost_shard1_replica_n7",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10074_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node8":{           "core":"testNodeLost_shard1_replica_n8",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10085_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node9":{           "core":"testNodeLost_shard1_replica_n9",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10056_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node2":{           "core":"testNodeLost_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10069_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node3":{           "core":"testNodeLost_shard1_replica_n3",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10064_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node4":{           "core":"testNodeLost_shard1_replica_n4",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10098_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node5":{           "core":"testNodeLost_shard1_replica_n5",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10075_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"80000000-8ccbffff",       "state":"active"},     "shard6":{       "replicas":{         "core_node60":{           "core":"testNodeLost_shard6_replica_n60",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10073_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node52":{           "core":"testNodeLost_shard6_replica_n52",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10021_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node51":{           "core":"testNodeLost_shard6_replica_n51",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10096_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node54":{           "core":"testNodeLost_shard6_replica_n54",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10037_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node53":{           "core":"testNodeLost_shard6_replica_n53",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10060_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node56":{           "core":"testNodeLost_shard6_replica_n56",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10031_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node55":{           "core":"testNodeLost_shard6_replica_n55",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10055_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node58":{           "core":"testNodeLost_shard6_replica_n58",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10059_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node57":{           "core":"testNodeLost_shard6_replica_n57",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10091_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node59":{           "core":"testNodeLost_shard6_replica_n59",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10015_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"c0000000-cccbffff",       "state":"active"},     "shard7":{       "replicas":{         "core_node70":{           "core":"testNodeLost_shard7_replica_n70",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10012_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node61":{           "core":"testNodeLost_shard7_replica_n61",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10089_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node63":{           "core":"testNodeLost_shard7_replica_n63",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10068_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node62":{           "core":"testNodeLost_shard7_replica_n62",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10067_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node65":{           "core":"testNodeLost_shard7_replica_n65",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10018_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node64":{           "core":"testNodeLost_shard7_replica_n64",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10065_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node67":{           "core":"testNodeLost_shard7_replica_n67",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10093_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node66":{           "core":"testNodeLost_shard7_replica_n66",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10009_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node69":{           "core":"testNodeLost_shard7_replica_n69",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10006_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node68":{           "core":"testNodeLost_shard7_replica_n68",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10039_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"cccc0000-d998ffff",       "state":"active"},     "shard8":{       "replicas":{         "core_node80":{           "core":"testNodeLost_shard8_replica_n80",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10047_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node72":{           "core":"testNodeLost_shard8_replica_n72",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10094_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node71":{           "core":"testNodeLost_shard8_replica_n71",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10025_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node74":{           "core":"testNodeLost_shard8_replica_n74",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10041_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node73":{           "core":"testNodeLost_shard8_replica_n73",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10004_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node76":{           "core":"testNodeLost_shard8_replica_n76",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10063_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node75":{           "core":"testNodeLost_shard8_replica_n75",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10070_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node78":{           "core":"testNodeLost_shard8_replica_n78",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10092_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node77":{           "core":"testNodeLost_shard8_replica_n77",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10052_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node79":{           "core":"testNodeLost_shard8_replica_n79",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10090_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"d9990000-e665ffff",       "state":"active"},     "shard9":{       "replicas":{         "core_node90":{           "core":"testNodeLost_shard9_replica_n90",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10084_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node81":{           "core":"testNodeLost_shard9_replica_n81",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10072_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node204":{           "node_name":"127.0.0.1:10056_solr",           "core":"testNodeLost_shard9_replica_n203",           "state":"active",           "INDEX.sizeInBytes":10240,           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6},         "core_node83":{           "core":"testNodeLost_shard9_replica_n83",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10066_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node82":{           "core":"testNodeLost_shard9_replica_n82",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10051_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node85":{           "core":"testNodeLost_shard9_replica_n85",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10053_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node84":{           "core":"testNodeLost_shard9_replica_n84",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10034_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node87":{           "core":"testNodeLost_shard9_replica_n87",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10014_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node86":{           "core":"testNodeLost_shard9_replica_n86",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10071_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node88":{           "core":"testNodeLost_shard9_replica_n88",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10097_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"e6660000-f332ffff",       "state":"active"}}}}, last coll state: DocCollection(testNodeLost//clusterstate.json/234)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"5",   "autoAddReplicas":"false",   "nrtReplicas":"10",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard20":{       "replicas":{         "core_node191":{           "core":"testNodeLost_shard20_replica_n191",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10000_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node200":{           "core":"testNodeLost_shard20_replica_n200",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10027_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node199":{           "core":"testNodeLost_shard20_replica_n199",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10058_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node198":{           "core":"testNodeLost_shard20_replica_n198",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10076_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node197":{           "core":"testNodeLost_shard20_replica_n197",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10028_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node196":{           "

[...truncated too long message...]

s.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1921051582
     [copy] Copying 239 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1921051582
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1921051582

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 33ms :: artifacts dl 4ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 502 minutes 43 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 341 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/341/

1 tests failed.
FAILED:  org.apache.solr.handler.component.TestDistributedStatsComponentCardinality.test

Error Message:
Error from server at http://127.0.0.1:40678//collection1: ERROR adding document SolrInputDocument(fields: [id=1876, int_i=1876, int_i_prehashed_l=-4301671625203428192, long_l=5777535722799, long_l_prehashed_l=8804101326278830356, string_s=s5777535722799, string_s_prehashed_l=4022662755234983409, _version_=1613820875930337280])

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:40678//collection1: ERROR adding document SolrInputDocument(fields: [id=1876, int_i=1876, int_i_prehashed_l=-4301671625203428192, long_l=5777535722799, long_l_prehashed_l=8804101326278830356, string_s=s5777535722799, string_s_prehashed_l=4022662755234983409, _version_=1613820875930337280])
        at __randomizedtesting.SeedInfo.seed([CCA4941EC30785C:849E769B42CC15A4]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
        at org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:483)
        at org.apache.solr.handler.component.TestDistributedStatsComponentCardinality.buildIndex(TestDistributedStatsComponentCardinality.java:89)
        at org.apache.solr.handler.component.TestDistributedStatsComponentCardinality.test(TestDistributedStatsComponentCardinality.java:106)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 15134 lines...]
   [junit4] Suite: org.apache.solr.handler.component.TestDistributedStatsComponentCardinality
   [junit4]   2> 4446434 INFO  (SUITE-TestDistributedStatsComponentCardinality-seed#[CCA4941EC30785C]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/init-core-data-001
   [junit4]   2> 4446435 WARN  (SUITE-TestDistributedStatsComponentCardinality-seed#[CCA4941EC30785C]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=351 numCloses=351
   [junit4]   2> 4446435 INFO  (SUITE-TestDistributedStatsComponentCardinality-seed#[CCA4941EC30785C]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 4446438 INFO  (SUITE-TestDistributedStatsComponentCardinality-seed#[CCA4941EC30785C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-9062)
   [junit4]   2> 4446438 INFO  (SUITE-TestDistributedStatsComponentCardinality-seed#[CCA4941EC30785C]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 4446646 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/control/cores/collection1
   [junit4]   2> 4446648 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 4446649 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4446649 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4446649 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4446649 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@44d23968{/,null,AVAILABLE}
   [junit4]   2> 4446650 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@105f2395{HTTP/1.1,[http/1.1]}{127.0.0.1:40678}
   [junit4]   2> 4446650 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server Started @4446833ms
   [junit4]   2> 4446650 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=40678, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/control/cores}
   [junit4]   2> 4446650 ERROR (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4446650 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4446651 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 4446651 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 4446651 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4446651 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T04:31:38.412Z
   [junit4]   2> 4446652 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/control/solr.xml
   [junit4]   2> 4446656 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4446656 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4446658 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4446910 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4446910 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4446939 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4446955 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4446955 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4446958 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/control/cores
   [junit4]   2> 4446958 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 4446963 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 4447013 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 4447040 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 4447164 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4447173 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 4447174 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4447174 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/control/cores/collection1/data/]
   [junit4]   2> 4447177 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@f25e62b
   [junit4]   2> 4447276 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4447276 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4447277 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4447278 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4447279 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.32303790353978024]
   [junit4]   2> 4447281 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@54bb7aed[collection1] main]
   [junit4]   2> 4447281 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 4447290 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4447323 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4447323 INFO  (searcherExecutor-13795-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 4447323 INFO  (coreLoadExecutor-13794-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613820853311504384
   [junit4]   2> 4447324 INFO  (searcherExecutor-13795-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 4447324 INFO  (searcherExecutor-13795-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 4447324 INFO  (searcherExecutor-13795-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 4447324 INFO  (searcherExecutor-13795-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 4447324 INFO  (searcherExecutor-13795-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 4447325 INFO  (searcherExecutor-13795-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 4447325 INFO  (searcherExecutor-13795-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 4447325 INFO  (searcherExecutor-13795-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 4447326 INFO  (searcherExecutor-13795-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@54bb7aed[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4447502 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard0/cores/collection1
   [junit4]   2> 4447503 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 4447504 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4447504 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4447504 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 4447505 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@ed33edb{/,null,AVAILABLE}
   [junit4]   2> 4447505 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@223acef4{HTTP/1.1,[http/1.1]}{127.0.0.1:45442}
   [junit4]   2> 4447505 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server Started @4447688ms
   [junit4]   2> 4447505 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=45442, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard0/cores}
   [junit4]   2> 4447505 ERROR (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4447505 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4447506 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 4447506 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 4447506 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4447506 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T04:31:39.267Z
   [junit4]   2> 4447506 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard0/solr.xml
   [junit4]   2> 4447511 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4447511 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4447512 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4447669 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4447669 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4447700 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4447718 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4447718 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4447721 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard0/cores
   [junit4]   2> 4447721 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 4447722 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 4447775 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 4447801 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 4447927 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4447936 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 4447937 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4447937 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard0/cores/collection1/data/]
   [junit4]   2> 4447940 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7047f784
   [junit4]   2> 4448031 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4448031 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4448033 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4448033 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4448034 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.32303790353978024]
   [junit4]   2> 4448036 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@77df9589[collection1] main]
   [junit4]   2> 4448037 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 4448045 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4448078 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4448078 INFO  (searcherExecutor-13809-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 4448078 INFO  (coreLoadExecutor-13808-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613820854103179264
   [junit4]   2> 4448078 INFO  (searcherExecutor-13809-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 4448078 INFO  (searcherExecutor-13809-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 4448078 INFO  (searcherExecutor-13809-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 4448079 INFO  (searcherExecutor-13809-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 4448079 INFO  (searcherExecutor-13809-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 4448079 INFO  (searcherExecutor-13809-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 4448080 INFO  (searcherExecutor-13809-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 4448080 INFO  (searcherExecutor-13809-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 4448081 INFO  (searcherExecutor-13809-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@77df9589[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4448265 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard1/cores/collection1
   [junit4]   2> 4448266 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 4448267 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4448267 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4448267 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4448268 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@60a6bf01{/,null,AVAILABLE}
   [junit4]   2> 4448268 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@182f9f4{HTTP/1.1,[http/1.1]}{127.0.0.1:38826}
   [junit4]   2> 4448268 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server Started @4448451ms
   [junit4]   2> 4448268 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=38826, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard1/cores}
   [junit4]   2> 4448269 ERROR (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4448269 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4448269 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 4448269 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 4448269 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4448269 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T04:31:40.030Z
   [junit4]   2> 4448269 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard1/solr.xml
   [junit4]   2> 4448274 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4448274 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4448276 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4448521 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4448521 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4448551 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4448569 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4448570 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4448573 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard1/cores
   [junit4]   2> 4448573 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 4448574 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 4448620 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 4448669 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 4448789 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4448798 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 4448798 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4448799 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard1/cores/collection1/data/]
   [junit4]   2> 4448802 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5d82cf7a
   [junit4]   2> 4448889 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4448889 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4448891 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4448891 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4448893 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.32303790353978024]
   [junit4]   2> 4448894 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@75595ea9[collection1] main]
   [junit4]   2> 4448895 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 4448902 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4448936 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4448936 INFO  (searcherExecutor-13823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 4448936 INFO  (coreLoadExecutor-13822-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613820855002857472
   [junit4]   2> 4448937 INFO  (searcherExecutor-13823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 4448937 INFO  (searcherExecutor-13823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 4448937 INFO  (searcherExecutor-13823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 4448937 INFO  (searcherExecutor-13823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 4448937 INFO  (searcherExecutor-13823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 4448938 INFO  (searcherExecutor-13823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 4448938 INFO  (searcherExecutor-13823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 4448938 INFO  (searcherExecutor-13823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 4448940 INFO  (searcherExecutor-13823-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@75595ea9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4449628 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard2/cores/collection1
   [junit4]   2> 4449630 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 4449631 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4449631 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4449631 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4449631 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@14d272bf{/,null,AVAILABLE}
   [junit4]   2> 4449632 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@74d08891{HTTP/1.1,[http/1.1]}{127.0.0.1:39928}
   [junit4]   2> 4449632 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server Started @4449815ms
   [junit4]   2> 4449632 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=39928, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard2/cores}
   [junit4]   2> 4449632 ERROR (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4449632 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4449632 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 4449632 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 4449632 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4449632 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T04:31:41.393Z
   [junit4]   2> 4449633 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard2/solr.xml
   [junit4]   2> 4449638 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4449638 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4449639 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4449767 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4449767 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4449799 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4449817 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4449818 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4449821 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard2/cores
   [junit4]   2> 4449821 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 4449822 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 4449869 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 4449902 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 4450056 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4450066 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 4450067 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4450067 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard2/cores/collection1/data/]
   [junit4]   2> 4450071 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@43ad826
   [junit4]   2> 4450205 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4450205 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4450207 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4450207 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4450209 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.32303790353978024]
   [junit4]   2> 4450210 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@20133d3d[collection1] main]
   [junit4]   2> 4450212 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 4450224 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4450288 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4450288 INFO  (searcherExecutor-13837-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 4450288 INFO  (coreLoadExecutor-13836-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613820856420532224
   [junit4]   2> 4450289 INFO  (searcherExecutor-13837-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 4450289 INFO  (searcherExecutor-13837-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 4450289 INFO  (searcherExecutor-13837-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 4450289 INFO  (searcherExecutor-13837-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 4450290 INFO  (searcherExecutor-13837-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 4450290 INFO  (searcherExecutor-13837-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 4450290 INFO  (searcherExecutor-13837-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 4450291 INFO  (searcherExecutor-13837-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 4450293 INFO  (searcherExecutor-13837-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@20133d3d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4450474 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard3/cores/collection1
   [junit4]   2> 4450476 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 4450496 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4450496 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4450496 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 4450496 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@10a0685e{/,null,AVAILABLE}
   [junit4]   2> 4450497 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7954e400{HTTP/1.1,[http/1.1]}{127.0.0.1:40575}
   [junit4]   2> 4450497 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server Started @4450680ms
   [junit4]   2> 4450497 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=40575, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard3/cores}
   [junit4]   2> 4450497 ERROR (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4450497 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4450497 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 4450497 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 4450497 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4450497 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T04:31:42.258Z
   [junit4]   2> 4450498 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard3/solr.xml
   [junit4]   2> 4450503 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4450503 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4450504 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4450673 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4450673 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4450706 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4450725 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4450726 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4450729 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard3/cores
   [junit4]   2> 4450729 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 4450730 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 4450783 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 4450808 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 4450974 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4450983 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 4450984 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4450984 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard3/cores/collection1/data/]
   [junit4]   2> 4450988 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5588b86
   [junit4]   2> 4451129 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4451129 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4451130 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4451131 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4451132 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.32303790353978024]
   [junit4]   2> 4451134 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@733d2338[collection1] main]
   [junit4]   2> 4451135 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 4451145 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4451182 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4451182 INFO  (searcherExecutor-13851-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 4451182 INFO  (searcherExecutor-13851-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 4451182 INFO  (searcherExecutor-13851-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 4451183 INFO  (searcherExecutor-13851-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 4451183 INFO  (searcherExecutor-13851-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 4451183 INFO  (searcherExecutor-13851-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 4451183 INFO  (searcherExecutor-13851-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 4451184 INFO  (searcherExecutor-13851-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 4451184 INFO  (searcherExecutor-13851-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 4451186 INFO  (searcherExecutor-13851-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@733d2338[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4451186 INFO  (coreLoadExecutor-13850-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613820857362153472
   [junit4]   2> 4451375 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard4/cores/collection1
   [junit4]   2> 4451377 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 4451378 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4451378 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4451378 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 4451378 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@15b8a043{/,null,AVAILABLE}
   [junit4]   2> 4451379 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4653d7c7{HTTP/1.1,[http/1.1]}{127.0.0.1:37574}
   [junit4]   2> 4451379 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server Started @4451562ms
   [junit4]   2> 4451379 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=37574, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard4/cores}
   [junit4]   2> 4451379 ERROR (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4451379 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4451379 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 4451379 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 4451379 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4451379 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T04:31:43.140Z
   [junit4]   2> 4451380 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard4/solr.xml
   [junit4]   2> 4451385 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4451385 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4451386 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4451673 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4451673 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4451708 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4451743 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4451743 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4451746 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard4/cores
   [junit4]   2> 4451746 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 4451747 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 4451796 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 4451822 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 4452003 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4452012 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 4452013 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4452013 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard4/cores/collection1/data/]
   [junit4]   2> 4452016 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2dde268c
   [junit4]   2> 4452104 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4452104 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4452105 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4452105 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4452107 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.32303790353978024]
   [junit4]   2> 4452108 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3a59b452[collection1] main]
   [junit4]   2> 4452109 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 4452117 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4452151 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 4452151 INFO  (searcherExecutor-13865-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 4452151 INFO  (coreLoadExecutor-13864-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613820858374029312
   [junit4]   2> 4452152 INFO  (searcherExecutor-13865-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 4452152 INFO  (searcherExecutor-13865-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 4452152 INFO  (searcherExecutor-13865-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 4452152 INFO  (searcherExecutor-13865-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 4452152 INFO  (searcherExecutor-13865-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 4452153 INFO  (searcherExecutor-13865-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 4452153 INFO  (searcherExecutor-13865-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 4452153 INFO  (searcherExecutor-13865-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 4452155 INFO  (searcherExecutor-13865-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3a59b452[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4452854 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard5/cores/collection1
   [junit4]   2> 4452855 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 4452857 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 4452857 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 4452857 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 4452857 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3713a62f{/,null,AVAILABLE}
   [junit4]   2> 4452858 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@27ba48d9{HTTP/1.1,[http/1.1]}{127.0.0.1:44400}
   [junit4]   2> 4452858 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.e.j.s.Server Started @4453041ms
   [junit4]   2> 4452858 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=44400, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard5/cores}
   [junit4]   2> 4452858 ERROR (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4452858 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 4452858 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 4452858 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 4452858 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4452858 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T04:31:44.619Z
   [junit4]   2> 4452859 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard5/solr.xml
   [junit4]   2> 4452864 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 4452864 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 4452865 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 4452948 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 4452948 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 4452984 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4453005 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4453006 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4453009 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard5/cores
   [junit4]   2> 4453009 INFO  (TEST-TestDistributedStatsComponentCardinality.test-seed#[CCA4941EC30785C]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 4453010 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 4453057 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 4453114 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 4453262 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 4453272 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1, trusted=true
   [junit4]   2> 4453273 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f58b1b6
   [junit4]   2> 4453273 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.component.TestDistributedStatsComponentCardinality_CCA4941EC30785C-001/tempDir-001/shard5/cores/collection1/data/]
   [junit4]   2> 4453277 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@16e20a2d
   [junit4]   2> 4453366 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 4453366 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4453368 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4453368 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4453369 INFO  (coreLoadExecutor-13878-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeFor

[...truncated too long message...]

x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null52758482
     [copy] Copying 239 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null52758482
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null52758482

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 43ms :: artifacts dl 2ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 368 minutes 15 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 342 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/342/

11 tests failed.
FAILED:  org.apache.lucene.document.TestLatLonLineShapeQueries.testRandomBig

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([1EECD2878E39E4FB]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.document.TestLatLonLineShapeQueries

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([1EECD2878E39E4FB]:0)


FAILED:  org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes

Error Message:
[127.0.0.1:39864_solr, 127.0.0.1:33769_solr] expected:<0> but was:<2>

Stack Trace:
java.lang.AssertionError: [127.0.0.1:39864_solr, 127.0.0.1:33769_solr] expected:<0> but was:<2>
        at __randomizedtesting.SeedInfo.seed([F44FD0660F7C220D:900F050AB4B2AA1C]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes(CollectionsAPIDistributedZkTest.java:351)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:44058_solr has 3 replicas. Expected num replicas : 2. state:  DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/13)={   "pullReplicas":0,   "replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node62":{           "core":"hdfsbackuprestore_restored_shard2_replica_n61",           "base_url":"http://127.0.0.1:44058/solr",           "node_name":"127.0.0.1:44058_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1539172459331507866"},     "shard1_1":{       "range":"c0000000-ffffffff",       "state":"active",       "replicas":{"core_node64":{           "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",           "base_url":"http://127.0.0.1:44058/solr",           "node_name":"127.0.0.1:44058_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1539172459331539813"},     "shard1_0":{       "range":"80000000-bfffffff",       "state":"active",       "replicas":{"core_node66":{           "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",           "base_url":"http://127.0.0.1:44058/solr",           "node_name":"127.0.0.1:44058_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"}},       "stateTimestamp":"1539172459331561671"}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"-1",   "autoAddReplicas":"false",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:44058_solr has 3 replicas. Expected num replicas : 2. state:
DocCollection(hdfsbackuprestore_restored//collections/hdfsbackuprestore_restored/state.json/13)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node62":{
          "core":"hdfsbackuprestore_restored_shard2_replica_n61",
          "base_url":"http://127.0.0.1:44058/solr",
          "node_name":"127.0.0.1:44058_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1539172459331507866"},
    "shard1_1":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{"core_node64":{
          "core":"hdfsbackuprestore_restored_shard1_1_replica_n63",
          "base_url":"http://127.0.0.1:44058/solr",
          "node_name":"127.0.0.1:44058_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1539172459331539813"},
    "shard1_0":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{"core_node66":{
          "core":"hdfsbackuprestore_restored_shard1_0_replica_n65",
          "base_url":"http://127.0.0.1:44058/solr",
          "node_name":"127.0.0.1:44058_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"}},
      "stateTimestamp":"1539172459331561671"}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"-1",
  "autoAddReplicas":"false",
  "nrtReplicas":1,
  "tlogReplicas":0}
        at __randomizedtesting.SeedInfo.seed([F44FD0660F7C220D:7C1BEFBCA1804FF5]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
        at java.util.HashMap.forEach(HashMap.java:1289)
        at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
        at org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
        at org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:213)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:35852_solr, 127.0.0.1:39793_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/21)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node4":{           "core":"testSimple1_shard1_replica_n2",           "base_url":"https://127.0.0.1:35852/solr",           "node_name":"127.0.0.1:35852_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node12":{           "core":"testSimple1_shard1_replica_n11",           "base_url":"https://127.0.0.1:35852/solr",           "node_name":"127.0.0.1:35852_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "core":"testSimple1_shard2_replica_n5",           "base_url":"https://127.0.0.1:35852/solr",           "node_name":"127.0.0.1:35852_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node10":{           "core":"testSimple1_shard2_replica_n9",           "base_url":"https://127.0.0.1:38785/solr",           "node_name":"127.0.0.1:38785_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:35852_solr, 127.0.0.1:39793_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/21)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"testSimple1_shard1_replica_n2",
          "base_url":"https://127.0.0.1:35852/solr",
          "node_name":"127.0.0.1:35852_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"https://127.0.0.1:35852/solr",
          "node_name":"127.0.0.1:35852_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "core":"testSimple1_shard2_replica_n5",
          "base_url":"https://127.0.0.1:35852/solr",
          "node_name":"127.0.0.1:35852_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"https://127.0.0.1:38785/solr",
          "node_name":"127.0.0.1:38785_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([F44FD0660F7C220D:CCFCF498288FF6DC]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:36728_solr, 127.0.0.1:38445_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/17)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node3/data/",           "base_url":"http://127.0.0.1:36728/solr",           "node_name":"127.0.0.1:36728_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node3/data/tlog",           "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",           "state":"active"},         "core_node5":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node5/data/",           "base_url":"http://127.0.0.1:36728/solr",           "node_name":"127.0.0.1:36728_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node5/data/tlog",           "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",           "state":"active",           "leader":"true"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node7/data/",           "base_url":"http://127.0.0.1:36728/solr",           "node_name":"127.0.0.1:36728_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node7/data/tlog",           "core":"testSimple1_shard2_replica_n4",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node8":{           "dataDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node8/data/",           "base_url":"http://127.0.0.1:37785/solr",           "node_name":"127.0.0.1:37785_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node8/data/tlog",           "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",           "state":"down"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:36728_solr, 127.0.0.1:38445_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/17)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node3/data/",
          "base_url":"http://127.0.0.1:36728/solr",
          "node_name":"127.0.0.1:36728_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active"},
        "core_node5":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node5/data/",
          "base_url":"http://127.0.0.1:36728/solr",
          "node_name":"127.0.0.1:36728_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node5/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node7/data/",
          "base_url":"http://127.0.0.1:36728/solr",
          "node_name":"127.0.0.1:36728_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n4",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node8":{
          "dataDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node8/data/",
          "base_url":"http://127.0.0.1:37785/solr",
          "node_name":"127.0.0.1:37785_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://lucene2-us-west.apache.org:43758/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([F44FD0660F7C220D:CCFCF498288FF6DC]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimComputePlanAction.testNodeWithMultipleReplicasLost

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([F44FD0660F7C220D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestSimComputePlanAction

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([F44FD0660F7C220D]:0)


FAILED:  org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([F44FD0660F7C220D:C7FDF8A202CBF8BA]:0)
        at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
        at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
        at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
        at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
        at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
        at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.addPosition(Lucene50PostingsWriter.java:291)
        at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:156)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
        at org.apache.lucene.codecs.asserting.AssertingPostingsFormat$AssertingFieldsConsumer.write(AssertingPostingsFormat.java:137)
        at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
        at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2170)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5103)
        at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1302)
        at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1272)
        at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:159)
        at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:65)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([D9F088C6BB00E44C:711E4F5545499040]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests(CloudSolrClientTest.java:550)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest

Error Message:
Response was not received from shards on a single node

Stack Trace:
java.lang.AssertionError: Response was not received from shards on a single node
        at __randomizedtesting.SeedInfo.seed([D9F088C6BB00E44C:253D107F45D773B6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.queryWithShardsPreferenceRules(CloudSolrClientTest.java:471)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest(CloudSolrClientTest.java:420)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 10533 lines...]
   [junit4] Suite: org.apache.lucene.document.TestLatLonLineShapeQueries
   [junit4]   2> lis 10, 2018 6:39:36 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.document.TestLatLonLineShapeQueries
   [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:637)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:494)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:269)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:394)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    2) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    3) Thread[id=11, name=SUITE-TestLatLonLineShapeQueries-seed#[1EECD2878E39E4FB], state=RUNNABLE, group=TGRP-TestLatLonLineShapeQueries]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:706)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:627)
   [junit4]   2>    4) Thread[id=12, name=TEST-TestLatLonLineShapeQueries.testRandomBig-seed#[1EECD2878E39E4FB], state=RUNNABLE, group=TGRP-TestLatLonLineShapeQueries]
   [junit4]   2>         at sun.nio.ch.NativeThread.current(Native Method)
   [junit4]   2>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
   [junit4]   2>         at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)
   [junit4]   2>         at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterFileChannel.read(FilterFileChannel.java:111)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterFileChannel.read(FilterFileChannel.java:111)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterFileChannel.read(FilterFileChannel.java:111)
   [junit4]   2>         at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
   [junit4]   2>         at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
   [junit4]   2>         at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
   [junit4]   2>         at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextUtil.readLine(SimpleTextUtil.java:59)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.readLine(SimpleTextBKDReader.java:407)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.visitDocValues(SimpleTextBKDReader.java:165)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:252)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:276)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:276)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:276)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:276)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:99)
   [junit4]   2>         at org.apache.lucene.document.LatLonShapeQuery$RelationScorerSupplier.getScorer(LatLonShapeQuery.java:369)
   [junit4]   2>         at org.apache.lucene.document.LatLonShapeQuery$1$4.get(LatLonShapeQuery.java:211)
   [junit4]   2>         at org.apache.lucene.document.LatLonShapeQuery$1.scorer(LatLonShapeQuery.java:261)
   [junit4]   2>         at org.apache.lucene.search.Weight.bulkScorer(Weight.java:177)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:832)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:88)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:667)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
   [junit4]   2>         at org.apache.lucene.document.BaseLatLonShapeTestCase.verifyRandomPolygonQueries(BaseLatLonShapeTestCase.java:349)
   [junit4]   2>         at org.apache.lucene.document.BaseLatLonShapeTestCase.verify(BaseLatLonShapeTestCase.java:199)
   [junit4]   2>         at org.apache.lucene.document.BaseLatLonShapeTestCase.doTestRandom(BaseLatLonShapeTestCase.java:172)
   [junit4]   2>         at org.apache.lucene.document.BaseLatLonShapeTestCase.testRandomBig(BaseLatLonShapeTestCase.java:148)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLatLonLineShapeQueries -Dtests.method=testRandomBig -Dtests.seed=1EECD2878E39E4FB -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=Pacific/Palau -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   7198s J2 | TestLatLonLineShapeQueries.testRandomBig <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1EECD2878E39E4FB]:0)
   [junit4]   2> lis 10, 2018 6:39:36 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> lis 10, 2018 6:39:56 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.document.TestLatLonLineShapeQueries:
   [junit4]   2>    1) Thread[id=12, name=TEST-TestLatLonLineShapeQueries.testRandomBig-seed#[1EECD2878E39E4FB], state=RUNNABLE, group=TGRP-TestLatLonLineShapeQueries]
   [junit4]   2>         at sun.nio.ch.NativeThread.current(Native Method)
   [junit4]   2>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
   [junit4]   2>         at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)
   [junit4]   2>         at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterFileChannel.read(FilterFileChannel.java:111)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterFileChannel.read(FilterFileChannel.java:111)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterFileChannel.read(FilterFileChannel.java:111)
   [junit4]   2>         at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
   [junit4]   2>         at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
   [junit4]   2>         at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
   [junit4]   2>         at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:140)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextUtil.readLine(SimpleTextUtil.java:59)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.readLine(SimpleTextBKDReader.java:407)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.visitDocValues(SimpleTextBKDReader.java:165)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:252)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:276)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:276)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:276)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:283)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:276)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:276)
   [junit4]   2>         at org.apache.lucene.codecs.simpletext.SimpleTextBKDReader.intersect(SimpleTextBKDReader.java:99)
   [junit4]   2>         at org.apache.lucene.document.LatLonShapeQuery$RelationScorerSupplier.getIntersectsScorer(LatLonShapeQuery.java:361)
   [junit4]   2>         at org.apache.lucene.document.LatLonShapeQuery$1$3.get(LatLonShapeQuery.java:194)
   [junit4]   2>         at org.apache.lucene.document.LatLonShapeQuery$1.scorer(LatLonShapeQuery.java:261)
   [junit4]   2>         at org.apache.lucene.search.Weight.bulkScorer(Weight.java:177)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:832)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:88)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:667)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
   [junit4]   2>         at org.apache.lucene.document.BaseLatLonShapeTestCase.verifyRandomPolygonQueries(BaseLatLonShapeTestCase.java:349)
   [junit4]   2>         at org.apache.lucene.document.BaseLatLonShapeTestCase.verify(BaseLatLonShapeTestCase.java:199)
   [junit4]   2>         at org.apache.lucene.document.BaseLatLonShapeTestCase.doTestRandom(BaseLatLonShapeTestCase.java:172)
   [junit4]   2>         at org.apache.lucene.document.BaseLatLonShapeTestCase.testRandomBig(BaseLatLonShapeTestCase.java:148)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> lis 10, 2018 6:39:56 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=12, name=TEST-TestLatLonLineShapeQueries.testRandomBig-seed#[1EECD2878E39E4FB], state=RUNNABLE, group=TGRP-TestLatLonLineShapeQueries]
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/sandbox/test/J2/temp/lucene.document.TestLatLonLineShapeQueries_1EECD2878E39E4FB-001
   [junit4]   2> Oct 10, 2018 9:39:56 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarity(queryNorm=true): {}, locale=hr, timezone=Pacific/Palau
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=311527432,total=483917824
   [junit4]   2> NOTE: All tests run in this JVM: [TestLatLonLineShapeQueries]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLatLonLineShapeQueries -Dtests.seed=1EECD2878E39E4FB -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=Pacific/Palau -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J2 | TestLatLonLineShapeQueries (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > at __randomizedtesting.SeedInfo.seed([1EECD2878E39E4FB]:0)
   [junit4] Completed [18/21 (1!)] on J2 in 7221.48s, 1 test, 2 errors <<< FAILURES!

[...truncated 2847 lines...]
   [junit4] Suite: org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDocTermOrdsUninvertLimit -Dtests.method=testTriggerUnInvertLimit -Dtests.seed=F44FD0660F7C220D -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ja-JP-u-ca-japanese-x-lvariant-JP -Dtests.timezone=SystemV/PST8PDT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   87.9s J0 | TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([F44FD0660F7C220D:C7FDF8A202CBF8BA]:0)
   [junit4]    > at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
   [junit4]    > at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
   [junit4]    > at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
   [junit4]    > at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
   [junit4]    > at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
   [junit4]    > at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.addPosition(Lucene50PostingsWriter.java:291)
   [junit4]    > at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:156)
   [junit4]    > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
   [junit4]    > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
   [junit4]    > at org.apache.lucene.codecs.asserting.AssertingPostingsFormat$AssertingFieldsConsumer.write(AssertingPostingsFormat.java:137)
   [junit4]    > at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]    > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
   [junit4]    > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
   [junit4]    > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
   [junit4]    > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]    > at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2170)
   [junit4]    > at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5103)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1302)
   [junit4]    > at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1272)
   [junit4]    > at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:159)
   [junit4]    > at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:65)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {field=DFR I(n)1}, locale=ja-JP-u-ca-japanese-x-lvariant-JP, timezone=SystemV/PST8PDT
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=59511056,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [TestCoreDiscovery, TestSerializedLuceneMatchVersion, TestSimNodeAddedTrigger, WrapperMergePolicyFactoryTest, TestAuthorizationFramework, QueryElevationComponentTest, LegacyCloudClusterPropTest, DirectoryFactoryTest, LIROnShardRestartTest, StatsComponentTest, TriggerSetPropertiesIntegrationTest, TermVectorComponentDistributedTest, SmileWriterTest, PreAnalyzedFieldTest, TestImplicitCoreProperties, CustomCollectionTest, TestDocTermOrdsUninvertLimit]
   [junit4] Completed [42/834 (1!)] on J0 in 89.11s, 1 test, 1 error <<< FAILURES!

[...truncated 791 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_F44FD0660F7C220D-001/init-core-data-001
   [junit4]   2> 2374562 WARN  (SUITE-AutoAddReplicasIntegrationTest-seed#[F44FD0660F7C220D]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=38 numCloses=38
   [junit4]   2> 2374563 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[F44FD0660F7C220D]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2374564 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[F44FD0660F7C220D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2374565 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[F44FD0660F7C220D]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2374574 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[F44FD0660F7C220D]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_F44FD0660F7C220D-001/tempDir-001
   [junit4]   2> 2374574 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[F44FD0660F7C220D]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2374575 INFO  (Thread-3912) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2374575 INFO  (Thread-3912) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2374577 ERROR (Thread-3912) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2374675 INFO  (SUITE-AutoAddReplicasIntegrationTest-seed#[F44FD0660F7C220D]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:33497
   [junit4]   2> 2374677 INFO  (zkConnectionManagerCallback-51651-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2374712 INFO  (jetty-launcher-51648-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2374733 INFO  (jetty-launcher-51648-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2374733 INFO  (jetty-launcher-51648-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2374733 INFO  (jetty-launcher-51648-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2374733 INFO  (jetty-launcher-51648-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@65226fd6{/solr,null,AVAILABLE}
   [junit4]   2> 2374734 INFO  (jetty-launcher-51648-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2374735 INFO  (jetty-launcher-51648-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@36fb3e74{SSL,[ssl, http/1.1]}{127.0.0.1:35852}
   [junit4]   2> 2374735 INFO  (jetty-launcher-51648-thread-1) [    ] o.e.j.s.Server Started @2374803ms
   [junit4]   2> 2374735 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35852}
   [junit4]   2> 2374735 ERROR (jetty-launcher-51648-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2374735 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2374735 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 2374735 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2374735 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2374735 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-10T11:04:48.365Z
   [junit4]   2> 2374735 INFO  (jetty-launcher-51648-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2374749 INFO  (jetty-launcher-51648-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2374749 INFO  (jetty-launcher-51648-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2374749 INFO  (jetty-launcher-51648-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2374749 INFO  (jetty-launcher-51648-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d6ac2c9{/solr,null,AVAILABLE}
   [junit4]   2> 2374765 INFO  (zkConnectionManagerCallback-51653-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2374766 INFO  (jetty-launcher-51648-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d70869a{SSL,[ssl, http/1.1]}{127.0.0.1:38785}
   [junit4]   2> 2374766 INFO  (jetty-launcher-51648-thread-2) [    ] o.e.j.s.Server Started @2374834ms
   [junit4]   2> 2374766 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38785}
   [junit4]   2> 2374766 ERROR (jetty-launcher-51648-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2374766 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2374766 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 2374767 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2374767 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2374767 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-10T11:04:48.397Z
   [junit4]   2> 2374773 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2374774 INFO  (jetty-launcher-51648-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2374774 INFO  (jetty-launcher-51648-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2374774 INFO  (jetty-launcher-51648-thread-3) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2374774 INFO  (jetty-launcher-51648-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@365088bb{/solr,null,AVAILABLE}
   [junit4]   2> 2374797 INFO  (jetty-launcher-51648-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@74aeeb1a{SSL,[ssl, http/1.1]}{127.0.0.1:39793}
   [junit4]   2> 2374797 INFO  (jetty-launcher-51648-thread-3) [    ] o.e.j.s.Server Started @2374865ms
   [junit4]   2> 2374797 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39793}
   [junit4]   2> 2374798 ERROR (jetty-launcher-51648-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2374798 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2374798 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 2374798 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2374798 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2374798 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-10T11:04:48.428Z
   [junit4]   2> 2374798 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2374798 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2374800 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2ac06a7a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2374800 INFO  (zkConnectionManagerCallback-51655-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2374801 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2374803 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2374803 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2374818 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2ac06a7a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2374878 INFO  (zkConnectionManagerCallback-51657-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2374879 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2374881 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2374881 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2374882 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2ac06a7a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2375304 INFO  (jetty-launcher-51648-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33497/solr
   [junit4]   2> 2375305 INFO  (zkConnectionManagerCallback-51661-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2375346 INFO  (zkConnectionManagerCallback-51663-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2375503 INFO  (jetty-launcher-51648-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33497/solr
   [junit4]   2> 2375503 INFO  (jetty-launcher-51648-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33497/solr
   [junit4]   2> 2375517 INFO  (zkConnectionManagerCallback-51671-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2375517 INFO  (zkConnectionManagerCallback-51673-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2375519 INFO  (zkConnectionManagerCallback-51675-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2375519 INFO  (zkConnectionManagerCallback-51677-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2375553 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39793_solr
   [junit4]   2> 2375554 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.c.Overseer Overseer (id=73467069524803592-127.0.0.1:39793_solr-n_0000000000) starting
   [junit4]   2> 2375572 INFO  (jetty-launcher-51648-thread-1) [n:127.0.0.1:35852_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35852_solr
   [junit4]   2> 2375599 INFO  (zkCallback-51676-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2375614 INFO  (zkCallback-51674-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2375614 INFO  (zkConnectionManagerCallback-51686-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2375615 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2375617 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33497/solr ready
   [junit4]   2> 2375618 INFO  (OverseerStateUpdate-73467069524803592-127.0.0.1:39793_solr-n_0000000000) [n:127.0.0.1:39793_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:39793_solr
   [junit4]   2> 2375648 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2375648 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39793_solr
   [junit4]   2> 2375649 INFO  (zkCallback-51676-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2375653 DEBUG (OverseerAutoScalingTriggerThread-73467069524803592-127.0.0.1:39793_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 2375675 INFO  (zkCallback-51674-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2375675 DEBUG (OverseerAutoScalingTriggerThread-73467069524803592-127.0.0.1:39793_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 2375676 DEBUG (OverseerAutoScalingTriggerThread-73467069524803592-127.0.0.1:39793_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 2375676 DEBUG (OverseerAutoScalingTriggerThread-73467069524803592-127.0.0.1:39793_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 2375677 INFO  (zkCallback-51685-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2375677 INFO  (zkConnectionManagerCallback-51691-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2375695 INFO  (jetty-launcher-51648-thread-1) [n:127.0.0.1:35852_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2375695 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2375696 INFO  (jetty-launcher-51648-thread-1) [n:127.0.0.1:35852_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33497/solr ready
   [junit4]   2> 2375696 INFO  (jetty-launcher-51648-thread-1) [n:127.0.0.1:35852_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2375742 DEBUG (OverseerAutoScalingTriggerThread-73467069524803592-127.0.0.1:39793_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:39793_solr, 127.0.0.1:35852_solr]
   [junit4]   2> 2375748 DEBUG (OverseerAutoScalingTriggerThread-73467069524803592-127.0.0.1:39793_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2375750 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ac06a7a
   [junit4]   2> 2375768 DEBUG (OverseerAutoScalingTriggerThread-73467069524803592-127.0.0.1:39793_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 2375895 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ac06a7a
   [junit4]   2> 2375895 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ac06a7a
   [junit4]   2> 2375896 INFO  (jetty-launcher-51648-thread-3) [n:127.0.0.1:39793_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasIntegrationTest_F44FD0660F7C220D-001/tempDir-001/node3/.
   [junit4]   2> 2375914 DEBUG (ScheduledTrigger-8867-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 2375918

[...truncated too long message...]

SolrCall.java:496)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   1> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
   [junit4]   1> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   1> at org.eclipse.jetty.server.Server.handle(Server.java:531)
   [junit4]   1> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
   [junit4]   1> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   1> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   1> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   1> at java.lang.Thread.run(Thread.java:748)
   [junit4]   1>
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrClientTest -Dtests.method=preferLocalShardsTest -Dtests.seed=D9F088C6BB00E44C -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=en-IN -Dtests.timezone=Etc/GMT-3 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 13.6s J2 | CloudSolrClientTest.preferLocalShardsTest <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Response was not received from shards on a single node
   [junit4]    > at __randomizedtesting.SeedInfo.seed([D9F088C6BB00E44C:253D107F45D773B6]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClientTest.queryWithShardsPreferenceRules(CloudSolrClientTest.java:471)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest(CloudSolrClientTest.java:420)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node4 rstate:down live:true
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node4 rstate:recovering live:true
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node4 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> -
   [junit4]   1> replica:core_node4 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> 12:40:23.721 [TEST-CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale-seed#[D9F088C6BB00E44C]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [stale_state_test_col] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:38653/solr/stale_state_test_col_shard1_replica_n1: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/stale_state_test_col_shard1_replica_n1/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/stale_state_test_col_shard1_replica_n1/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/stale_state_test_col_shard1_replica_n1/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 0
   [junit4]   1> 12:40:26.358 [TEST-CloudSolrClientTest.testShutdown-seed#[D9F088C6BB00E44C]-SendThread(ff01::114:33332)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   1> 12:40:27.459 [TEST-CloudSolrClientTest.testShutdown-seed#[D9F088C6BB00E44C]-SendThread(ff01::114:33332)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   1> 12:40:30.091 [SUITE-CloudSolrClientTest-seed#[D9F088C6BB00E44C]-worker] ERROR org.apache.zookeeper.server.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.impl.CloudSolrClientTest_D9F088C6BB00E44C-001
   [junit4]   2> Oct 10, 2018 12:40:30 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=FSTOrd50, a_t=FSTOrd50, title_s=FSTOrd50, id=PostingsFormat(name=LuceneVarGapDocFreqInterval), text=PostingsFormat(name=LuceneFixedGap)}, docValues:{_version_=DocValuesFormat(name=Lucene70), multiDefault=DocValuesFormat(name=Direct), a_t=DocValuesFormat(name=Direct), title_s=DocValuesFormat(name=Direct), intDefault=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Memory), text=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1523, maxMBSortInHeap=6.998132490999837, sim=RandomSimilarity(queryNorm=false): {}, locale=en-IN, timezone=Etc/GMT-3
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=142223080,total=450363392
   [junit4]   2> NOTE: All tests run in this JVM: [TestDelegationTokenRequest, TestBatchUpdate, RegressionEvaluatorTest, ConcurrentUpdateSolrClientMultiCollectionTest, TestSolrJsonWriter, HyperbolicSineEvaluatorTest, MultiplyEvaluatorTest, LessThanEqualToEvaluatorTest, TestEmbeddedSolrServer, DocumentAnalysisResponseTest, TestV1toV2ApiMapper, TestToleratedUpdateError, JDBCStreamTest, PowerEvaluatorTest, NormalizeEvaluatorTest, StreamDecoratorTest, ExclusiveOrEvaluatorTest, CosineEvaluatorTest, HttpSolrClientBuilderTest, TestNamedListCodec, TestClusteringResponse, SolrExampleJettyTest, CoalesceEvaluatorTest, UuidEvaluatorTest, TestDocumentObjectBinder, TestCollectionStateWatchers, LargeVolumeJettyTest, TestSolrProperties, BasicHttpSolrClientTest, CloudSolrClientTest]
   [junit4] Completed [137/157 (1!)] on J2 in 80.67s, 16 tests, 2 failures <<< FAILURES!

[...truncated 7967 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:651: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid5940.hprof

Total time: 592 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 343 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/343/

3 tests failed.
FAILED:  org.apache.solr.cloud.cdcr.CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates

Error Message:
Timeout while trying to assert number of documents @ source_collection

Stack Trace:
java.lang.AssertionError: Timeout while trying to assert number of documents @ source_collection
        at __randomizedtesting.SeedInfo.seed([746BA5538DE908D4:A762F54DC87A9443]:0)
        at org.apache.solr.cloud.cdcr.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:278)
        at org.apache.solr.cloud.cdcr.CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates(CdcrReplicationHandlerTest.java:234)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.AssertionError: expected:<180> but was:<190>
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.cdcr.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:269)
        ... 42 more


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferReplicaTypesTest

Error Message:
Could not load collection from ZK: replicaTypesTestColl

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: replicaTypesTestColl
        at __randomizedtesting.SeedInfo.seed([22FAD5837FEB937F:1910763518725E73]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1321)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:737)
        at org.apache.solr.common.cloud.ClusterState$CollectionRef.get(ClusterState.java:386)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1209)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:849)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.queryWithPreferReplicaTypes(CloudSolrClientTest.java:941)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferReplicaTypesTest(CloudSolrClientTest.java:892)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/replicaTypesTestColl/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:341)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:341)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1333)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1319)
        ... 49 more


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime

Error Message:
Error from server at http://127.0.0.1:34465/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason: <pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:34465/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([22FAD5837FEB937F:CC22AE12319446EC]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime(CloudSolrClientTest.java:146)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:34465/solr/collection1_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more




Build Log:
[...truncated 14167 lines...]
   [junit4] Suite: org.apache.solr.cloud.cdcr.CdcrReplicationHandlerTest
   [junit4]   2> 2257585 INFO  (SUITE-CdcrReplicationHandlerTest-seed#[746BA5538DE908D4]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/init-core-data-001
   [junit4]   2> 2257586 INFO  (SUITE-CdcrReplicationHandlerTest-seed#[746BA5538DE908D4]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2257598 INFO  (SUITE-CdcrReplicationHandlerTest-seed#[746BA5538DE908D4]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2257599 INFO  (SUITE-CdcrReplicationHandlerTest-seed#[746BA5538DE908D4]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /v/yu
   [junit4]   2> 2257619 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2257632 INFO  (Thread-27349) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2257632 INFO  (Thread-27349) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2257660 ERROR (Thread-27349) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2257732 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.ZkTestServer start zk server on port:37557
   [junit4]   2> 2257760 INFO  (zkConnectionManagerCallback-929-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2257796 INFO  (zkConnectionManagerCallback-931-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2257818 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2257819 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2257820 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2257821 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2257822 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2257823 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2257824 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2257825 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2257826 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2257827 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2257828 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2257830 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPartialReplicationAfterPeerSync
   [junit4]   2> 2258014 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2258056 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2258056 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2258056 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2258058 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@30d178e5{/v/yu,null,AVAILABLE}
   [junit4]   2> 2258059 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@35e0c235{HTTP/1.1,[http/1.1]}{127.0.0.1:37899}
   [junit4]   2> 2258059 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.Server Started @2258253ms
   [junit4]   2> 2258059 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard1, hostContext=/v/yu, hostPort=37899, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/jetty-001/cores}
   [junit4]   2> 2258059 ERROR (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2258059 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2258059 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 2258059 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2258059 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2258059 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-11T07:30:25.818Z
   [junit4]   2> 2258089 INFO  (zkConnectionManagerCallback-933-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2258091 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2258091 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/jetty-001/solr.xml
   [junit4]   2> 2258095 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2258095 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2258100 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2a245e60, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2258303 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37557/solr
   [junit4]   2> 2258318 INFO  (zkConnectionManagerCallback-937-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2258327 INFO  (zkConnectionManagerCallback-939-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2258466 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37899_v%2Fyu
   [junit4]   2> 2258476 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.Overseer Overseer (id=73471888858284036-127.0.0.1:37899_v%2Fyu-n_0000000000) starting
   [junit4]   2> 2258513 INFO  (zkConnectionManagerCallback-946-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2258515 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37557/solr ready
   [junit4]   2> 2258525 INFO  (OverseerStateUpdate-73471888858284036-127.0.0.1:37899_v%2Fyu-n_0000000000) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37899_v%2Fyu
   [junit4]   2> 2258547 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37899_v%2Fyu
   [junit4]   2> 2258568 INFO  (zkCallback-938-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2258572 INFO  (zkCallback-945-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2258594 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2258621 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a245e60
   [junit4]   2> 2258644 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a245e60
   [junit4]   2> 2258645 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a245e60
   [junit4]   2> 2258646 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/jetty-001/cores
   [junit4]   2> 2258865 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2258866 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2258866 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2258866 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2258866 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5e173d72{/v/yu,null,AVAILABLE}
   [junit4]   2> 2258867 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@33a7bcba{HTTP/1.1,[http/1.1]}{127.0.0.1:38687}
   [junit4]   2> 2258867 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.e.j.s.Server Started @2259061ms
   [junit4]   2> 2258867 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard2, hostContext=/v/yu, hostPort=38687, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/jetty-002/cores}
   [junit4]   2> 2258867 ERROR (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2258867 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2258867 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 2258867 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2258867 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2258867 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-11T07:30:26.626Z
   [junit4]   2> 2258896 INFO  (zkConnectionManagerCallback-949-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2258908 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2258908 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/jetty-002/solr.xml
   [junit4]   2> 2258911 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2258911 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2258912 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2a245e60, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2259091 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37557/solr
   [junit4]   2> 2259104 INFO  (zkConnectionManagerCallback-953-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2259107 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x105064b2c600007, likely client has closed socket
   [junit4]   2> 2259110 INFO  (zkConnectionManagerCallback-955-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2259120 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2259128 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2259128 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38687_v%2Fyu
   [junit4]   2> 2259132 INFO  (zkCallback-945-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2259132 INFO  (zkCallback-938-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2259153 INFO  (zkCallback-954-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2259158 INFO  (zkConnectionManagerCallback-962-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2259160 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2259161 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37557/solr ready
   [junit4]   2> 2259161 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2259201 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a245e60
   [junit4]   2> 2259228 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a245e60
   [junit4]   2> 2259233 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a245e60
   [junit4]   2> 2259234 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/jetty-002/cores
   [junit4]   2> 2259364 INFO  (zkConnectionManagerCallback-968-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2259368 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2259372 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37557/solr ready
   [junit4]   2> 2259393 INFO  (qtp1443446959-30675) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=tmp_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2259409 INFO  (OverseerThreadFactory-1526-thread-1-processing-n:127.0.0.1:37899_v%2Fyu) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.a.c.CreateCollectionCmd Create collection tmp_collection
   [junit4]   2> 2259523 WARN  (OverseerThreadFactory-1526-thread-1-processing-n:127.0.0.1:37899_v%2Fyu) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (tmp_collection) without cores.
   [junit4]   2> 2259529 INFO  (qtp1443446959-30675) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2259529 INFO  (qtp1443446959-30675) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=tmp_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&wt=javabin&version=2} status=0 QTime=136
   [junit4]   2> 2259530 INFO  (qtp1443446959-30671) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:37899_v%252Fyu&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2259549 INFO  (OverseerCollectionConfigSetProcessor-73471888858284036-127.0.0.1:37899_v%2Fyu-n_0000000000) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2259561 INFO  (qtp1443446959-30670) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2259568 INFO  (qtp600464345-30713) [n:127.0.0.1:38687_v%2Fyu    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2259570 INFO  (OverseerThreadFactory-1526-thread-2-processing-n:127.0.0.1:37899_v%2Fyu) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:37899_v%2Fyu for creating new replica of shard shard1 for collection tmp_collection
   [junit4]   2> 2259577 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n21&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2259577 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2260599 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 2260623 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.s.IndexSchema [tmp_collection_shard1_replica_n21] Schema name=test
   [junit4]   2> 2260827 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 2260882 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.CoreContainer Creating SolrCore 'tmp_collection_shard1_replica_n21' using configuration from collection tmp_collection, trusted=true
   [junit4]   2> 2260887 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.tmp_collection.shard1.replica_n21' (registry 'solr.core.tmp_collection.shard1.replica_n21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a245e60
   [junit4]   2> 2260888 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.SolrCore [[tmp_collection_shard1_replica_n21] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/jetty-001/cores/tmp_collection_shard1_replica_n21], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/jetty-001/cores/tmp_collection_shard1_replica_n21/data/]
   [junit4]   2> 2260892 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=19.1318359375, floorSegmentMB=1.0322265625, forceMergeDeletesPctAllowed=10.942490727582634, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=27.245467305135872
   [junit4]   2> 2261083 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 2261083 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2261084 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2261084 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2261086 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=45.61328125, floorSegmentMB=1.5107421875, forceMergeDeletesPctAllowed=23.016795121114367, segmentsPerTier=3.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.22084176171183573, deletesPctAllowed=28.68810347885577
   [junit4]   2> 2261086 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.s.SolrIndexSearcher Opening [Searcher@676f6b26[tmp_collection_shard1_replica_n21] main]
   [junit4]   2> 2261088 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2261088 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2261089 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2261102 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.h.CdcrBufferStateManager Created znode /collections/tmp_collection/cdcr/state/buffer
   [junit4]   2> 2261108 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.h.CdcrProcessStateManager Created znode /collections/tmp_collection/cdcr/state/process
   [junit4]   2> 2261139 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1614013298269749248
   [junit4]   2> 2261146 INFO  (searcherExecutor-1540-thread-1-processing-n:127.0.0.1:37899_v%2Fyu x:tmp_collection_shard1_replica_n21 c:tmp_collection s:shard1) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.SolrCore [tmp_collection_shard1_replica_n21] Registered new searcher Searcher@676f6b26[tmp_collection_shard1_replica_n21] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2261168 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tmp_collection/terms/shard1 to Terms{values={core_node22=0}, version=0}
   [junit4]   2> 2261171 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2261171 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2261171 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37899/v/yu/tmp_collection_shard1_replica_n21/
   [junit4]   2> 2261171 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2261171 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.SyncStrategy http://127.0.0.1:37899/v/yu/tmp_collection_shard1_replica_n21/ has no replicas
   [junit4]   2> 2261171 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2261174 INFO  (zkCallback-938-thread-1) [    ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ tmp_collection:shard1
   [junit4]   2> 2261177 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37899/v/yu/tmp_collection_shard1_replica_n21/ shard1
   [junit4]   2> 2261283 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2261286 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n21&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1709
   [junit4]   2> 2261291 INFO  (qtp1443446959-30671) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:37899_v%252Fyu&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1761
   [junit4]   2> 2261292 INFO  (qtp1443446959-30676) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:38687_v%252Fyu&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2261318 INFO  (OverseerCollectionConfigSetProcessor-73471888858284036-127.0.0.1:37899_v%2Fyu-n_0000000000) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2261338 INFO  (qtp1443446959-30670) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.tmp_collection.shard1.replica_n21:INDEX.sizeInBytes} status=0 QTime=4
   [junit4]   2> 2261350 INFO  (qtp1443446959-30673) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2261360 INFO  (qtp600464345-30709) [n:127.0.0.1:38687_v%2Fyu    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2261368 INFO  (qtp1443446959-30673) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.tmp_collection.shard1.replica_n21:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 2261369 INFO  (OverseerThreadFactory-1526-thread-3-processing-n:127.0.0.1:37899_v%2Fyu) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:38687_v%2Fyu for creating new replica of shard shard1 for collection tmp_collection
   [junit4]   2> 2261375 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu    x:tmp_collection_shard1_replica_n23] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n23&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2261485 INFO  (zkCallback-938-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2262400 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 2262423 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.s.IndexSchema [tmp_collection_shard1_replica_n23] Schema name=test
   [junit4]   2> 2262566 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 2262605 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.c.CoreContainer Creating SolrCore 'tmp_collection_shard1_replica_n23' using configuration from collection tmp_collection, trusted=true
   [junit4]   2> 2262605 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.tmp_collection.shard1.replica_n23' (registry 'solr.core.tmp_collection.shard1.replica_n23') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a245e60
   [junit4]   2> 2262606 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.c.SolrCore [[tmp_collection_shard1_replica_n23] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/jetty-002/cores/tmp_collection_shard1_replica_n23], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrReplicationHandlerTest_746BA5538DE908D4-001/jetty-002/cores/tmp_collection_shard1_replica_n23/data/]
   [junit4]   2> 2262635 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=19.1318359375, floorSegmentMB=1.0322265625, forceMergeDeletesPctAllowed=10.942490727582634, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=27.245467305135872
   [junit4]   2> 2262724 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 2262724 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2262725 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2262727 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2262728 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=45.61328125, floorSegmentMB=1.5107421875, forceMergeDeletesPctAllowed=23.016795121114367, segmentsPerTier=3.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.22084176171183573, deletesPctAllowed=28.68810347885577
   [junit4]   2> 2262730 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.s.SolrIndexSearcher Opening [Searcher@2335a388[tmp_collection_shard1_replica_n23] main]
   [junit4]   2> 2262731 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2262731 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2262733 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2262743 INFO  (searcherExecutor-1546-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.c.SolrCore [tmp_collection_shard1_replica_n23] Registered new searcher Searcher@2335a388[tmp_collection_shard1_replica_n23] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2262743 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1614013299951665152
   [junit4]   2> 2262755 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tmp_collection/terms/shard1 to Terms{values={core_node24=0, core_node22=0}, version=1}
   [junit4]   2> 2262756 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.c.ZkController Core needs to recover:tmp_collection_shard1_replica_n23
   [junit4]   2> 2262757 INFO  (qtp600464345-30708) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1  x:tmp_collection_shard1_replica_n23] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=tmp_collection_shard1_replica_n23&action=CREATE&collection=tmp_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1386
   [junit4]   2> 2262760 INFO  (qtp1443446959-30676) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:38687_v%252Fyu&action=ADDREPLICA&collection=tmp_collection&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1468
   [junit4]   2> 2262762 INFO  (OverseerCollectionConfigSetProcessor-73471888858284036-127.0.0.1:37899_v%2Fyu-n_0000000000) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2262764 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: tmp_collection failOnTimeout:true timeout (sec):330
   [junit4]   1> -
   [junit4]   1> replica:core_node22 rstate:active live:true
   [junit4]   1> replica:core_node24 rstate:down live:true
   [junit4]   2> 2262764 INFO  (updateExecutor-950-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 2262775 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 2262780 INFO  (qtp1443446959-30673) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1 r:core_node22 x:tmp_collection_shard1_replica_n21] o.a.s.c.S.Request [tmp_collection_shard1_replica_n21]  webapp=/v/yu path=/cdcr params={action=LASTPROCESSEDVERSION&wt=javabin&version=2} status=0 QTime=5
   [junit4]   2> 2262788 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy startupVersions is empty
   [junit4]   2> 2262790 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1 r:core_node22 x:tmp_collection_shard1_replica_n21] o.a.s.c.S.Request [tmp_collection_shard1_replica_n21]  webapp=/v/yu path=/admin/ping params={wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2262790 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1 r:core_node22 x:tmp_collection_shard1_replica_n21] o.a.s.c.S.Request [tmp_collection_shard1_replica_n21]  webapp=/v/yu path=/admin/ping params={wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2262791 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[tmp_collection_shard1_replica_n23]
   [junit4]   2> 2262791 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 2262791 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Publishing state of core [tmp_collection_shard1_replica_n23] as recovering, leader is [http://127.0.0.1:37899/v/yu/tmp_collection_shard1_replica_n21/] and I am [http://127.0.0.1:38687/v/yu/tmp_collection_shard1_replica_n23/]
   [junit4]   2> 2262793 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tmp_collection/terms/shard1 to Terms{values={core_node24_recovering=0, core_node24=0, core_node22=0}, version=2}
   [junit4]   2> 2262795 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:37899/v/yu]; [WaitForState: action=PREPRECOVERY&core=tmp_collection_shard1_replica_n21&nodeName=127.0.0.1:38687_v%252Fyu&coreNodeName=core_node24&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 2262796 INFO  (qtp1443446959-30673) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node24, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true, maxTime: 183 s
   [junit4]   2> 2262797 INFO  (qtp1443446959-30673) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=tmp_collection, shard=shard1, thisCore=tmp_collection_shard1_replica_n21, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:38687_v%2Fyu, coreNodeName=core_node24, onlyIfActiveCheckResult=false, nodeProps: core_node24:{"core":"tmp_collection_shard1_replica_n23","base_url":"http://127.0.0.1:38687/v/yu","node_name":"127.0.0.1:38687_v%2Fyu","state":"down","type":"NRT"}
   [junit4]   2> 2262895 INFO  (zkCallback-938-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2262895 INFO  (zkCallback-954-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2262896 INFO  (zkCallback-954-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   1> -
   [junit4]   1> replica:core_node22 rstate:active live:true
   [junit4]   1> replica:core_node24 rstate:recovering live:true
   [junit4]   2> 2263790 INFO  (qtp1443446959-30676) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1 r:core_node22 x:tmp_collection_shard1_replica_n21] o.a.s.c.S.Request [tmp_collection_shard1_replica_n21]  webapp=/v/yu path=/cdcr params={action=LASTPROCESSEDVERSION&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2263797 INFO  (qtp1443446959-30673) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=tmp_collection, shard=shard1, thisCore=tmp_collection_shard1_replica_n21, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:38687_v%2Fyu, coreNodeName=core_node24, onlyIfActiveCheckResult=false, nodeProps: core_node24:{"core":"tmp_collection_shard1_replica_n23","base_url":"http://127.0.0.1:38687/v/yu","node_name":"127.0.0.1:38687_v%2Fyu","state":"recovering","type":"NRT"}
   [junit4]   2> 2263797 INFO  (qtp1443446959-30673) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node24, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 2263797 INFO  (qtp1443446959-30673) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:38687_v%252Fyu&onlyIfLeaderActive=true&core=tmp_collection_shard1_replica_n21&coreNodeName=core_node24&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
   [junit4]   2> 2264299 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:37899/v/yu/tmp_collection_shard1_replica_n21/] - recoveringAfterStartup=[true]
   [junit4]   2> 2264299 WARN  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.u.PeerSyncWithLeader no frame of reference to tell if we've missed updates
   [junit4]   2> 2264300 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy PeerSync Recovery was not successful - trying replication.
   [junit4]   2> 2264300 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Starting Replication Recovery.
   [junit4]   2> 2264300 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Attempting to replicate from [http://127.0.0.1:37899/v/yu/tmp_collection_shard1_replica_n21/].
   [junit4]   2> 2264304 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1 r:core_node22 x:tmp_collection_shard1_replica_n21] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1614013301588492288,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2264304 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1 r:core_node22 x:tmp_collection_shard1_replica_n21] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2264304 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1 r:core_node22 x:tmp_collection_shard1_replica_n21] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2264304 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1 r:core_node22 x:tmp_collection_shard1_replica_n21] o.a.s.c.S.Request [tmp_collection_shard1_replica_n21]  webapp=/v/yu path=/update params={waitSearcher=true&openSearcher=false&commit=true&softCommit=false&commit_end_point=true&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 2264306 INFO  (qtp1443446959-30676) [n:127.0.0.1:37899_v%2Fyu c:tmp_collection s:shard1 r:core_node22 x:tmp_collection_shard1_replica_n21] o.a.s.c.S.Request [tmp_collection_shard1_replica_n21]  webapp=/v/yu path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 2264308 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 2264308 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 2264308 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 2264308 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 2264308 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 2264322 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.s.SolrIndexSearcher Opening [Searcher@3a5c7154[tmp_collection_shard1_replica_n23] main]
   [junit4]   2> 2264324 INFO  (searcherExecutor-1546-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.SolrCore [tmp_collection_shard1_replica_n23] Registered new searcher Searcher@3a5c7154[tmp_collection_shard1_replica_n23] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2264324 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 2264324 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Replication Recovery was successful.
   [junit4]   2> 2264324 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 2264325 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tmp_collection/terms/shard1 to Terms{values={core_node22=0, core_node24=0}, version=3}
   [junit4]   2> 2264326 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Updating version bucket highest from index after successful recovery.
   [junit4]   2> 2264326 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1614013301611560960
   [junit4]   2> 2264328 INFO  (recoveryExecutor-951-thread-1-processing-n:127.0.0.1:38687_v%2Fyu x:tmp_collection_shard1_replica_n23 c:tmp_collection s:shard1 r:core_node24) [n:127.0.0.1:38687_v%2Fyu c:tmp_collection s:shard1 r:core_node24 x:tmp_collection_shard1_replica_n23] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[true]
   [junit4]   2> 2264429 INFO  (zkCallback-938-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2264429 INFO  (zkCallback-954-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2264429 INFO  (zkCallback-954-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tmp_collection/state.json] for collection [tmp_collection] has occurred - updating... (live nodes size: [2])
   [junit4]   1> -
   [junit4]   1> replica:core_node22 rstate:active live:true
   [junit4]   1> replica:core_node24 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   2> 2264764 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: tmp_collection
   [junit4]   2> 2264772 INFO  (zkConnectionManagerCallback-974-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2264772 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2264774 INFO  (TEST-CdcrReplicationHandlerTest.testPartialReplicationAfterPeerSync-seed#[746BA5538DE908D4]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37557/solr ready
   [junit4]   2> 2264775 INFO  (qtp1443446959-30673) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=tmp_collection&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2264777 INFO  (OverseerThreadFactory-1526-thread-4-processing-n:127.0.0.1:37899_v%2Fyu) [n:127.0.0.1:37899_v%2Fyu    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true&deleteMetricsHistory=true, asyncId=null
   [junit4]   2> 2264779 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tmp_collection.shard1.replica_n21, tag=null
   [junit4]   2> 2264779 INFO  (qtp600464345-30712) [n:127.0.0.1:38687_v%2Fyu    x:tmp_collection_shard1_replica_n23] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tmp_collection.shard1.replica_n23, tag=null
   [junit4]   2> 2264779 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@297c401c: rootName = null, domain = solr.core.tmp_collection.shard1.replica_n21, service url = null, agent id = null] for registry solr.core.tmp_collection.shard1.replica_n21 / com.codahale.metrics.MetricRegistry@7ae6e313
   [junit4]   2> 2264779 INFO  (qtp600464345-30712) [n:127.0.0.1:38687_v%2Fyu    x:tmp_collection_shard1_replica_n23] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4e52bcf3: rootName = null, domain = solr.core.tmp_collection.shard1.replica_n23, service url = null, agent id = null] for registry solr.core.tmp_collection.shard1.replica_n23 / com.codahale.metrics.MetricRegistry@1dc23094
   [junit4]   2> 2264792 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.c.SolrCore [tmp_collection_shard1_replica_n21]  CLOSING SolrCore org.apache.solr.core.SolrCore@5d2c9ce0
   [junit4]   2> 2264792 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tmp_collection.shard1.replica_n21, tag=5d2c9ce0
   [junit4]   2> 2264792 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tmp_collection.shard1.leader, tag=5d2c9ce0
   [junit4]   2> 2264792 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.h.CdcrRequestHandler Solr core is being closed - shutting down CDCR handler @ tmp_collection:shard1
   [junit4]   2> 2264792 INFO  (qtp600464345-30712) [n:127.0.0.1:38687_v%2Fyu    x:tmp_collection_shard1_replica_n23] o.a.s.c.SolrCore [tmp_collection_shard1_replica_n23]  CLOSING SolrCore org.apache.solr.core.SolrCore@3a654a71
   [junit4]   2> 2264792 INFO  (qtp600464345-30712) [n:127.0.0.1:38687_v%2Fyu    x:tmp_collection_shard1_replica_n23] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tmp_collection.shard1.replica_n23, tag=3a654a71
   [junit4]   2> 2264792 INFO  (qtp600464345-30712) [n:127.0.0.1:38687_v%2Fyu    x:tmp_collection_shard1_replica_n23] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tmp_collection.shard1.leader, tag=3a654a71
   [junit4]   2> 2264792 INFO  (qtp600464345-30712) [n:127.0.0.1:38687_v%2Fyu    x:tmp_collection_shard1_replica_n23] o.a.s.h.CdcrRequestHandler Solr core is being closed - shutting down CDCR handler @ tmp_collection:shard1
   [junit4]   2> 2264807 WARN  (cdcr-update-log-synchronizer-1551-thread-1) [    ] o.a.s.h.CdcrUpdateLogSynchronizer Caught unexpected exception
   [junit4]   2> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:37899/v/yu/tmp_collection_shard1_replica_n21: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   2> <head>
   [junit4]   2> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   2> <title>Error 404 Can not find: /v/yu/tmp_collection_shard1_replica_n21/cdcr</title>
   [junit4]   2> </head>
   [junit4]   2> <body><h2>HTTP ERROR 404</h2>
   [junit4]   2> <p>Problem accessing /v/yu/tmp_collection_shard1_replica_n21/cdcr. Reason:
   [junit4]   2> <pre>    Can not find: /v/yu/tmp_collection_shard1_replica_n21/cdcr</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   2>
   [junit4]   2> </body>
   [junit4]   2> </html>
   [junit4]   2>
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) ~[java/:?]
   [junit4]   2> at org.apache.solr.handler.CdcrUpdateLogSynchronizer$UpdateLogSynchronisation.run(CdcrUpdateLogSynchronizer.java:147) [java/:?]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 2264809 INFO  (qtp1443446959-30674) [n:127.0.0.1:37899_v%2Fyu    x:tmp_collection_shard1_replica_n21] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tmp_collection/terms/shard1 to Terms{values={core_node24=0}, version=4}
   [junit4]   2> 2264810 WARN  (zkCallback-954-thread-2) [    ] o.a.s.c.LeaderElector
   [junit4]   2> org.apache.solr.common.SolrException: SolrCore not found:tmp_collection_shard1_replica_n23 in []
   [junit4]   2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:312) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56) ~[java/:?]
   [junit4]   2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.

[...truncated too long message...]

ngs.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null478639071
     [copy] Copying 239 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null478639071
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null478639071

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 82ms :: artifacts dl 13ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 545 minutes 44 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 344 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/344/

3 tests failed.
FAILED:  org.apache.lucene.index.TestIndexWriterDelete.testUpdatesOnDiskFull

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([B468FA9DABA4598]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterDelete

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([B468FA9DABA4598]:0)


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([951F32873E39A1C1:1D4B0D5D90C5CC39]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.MultiThreadedOCPTest.testFillWorkQueue(MultiThreadedOCPTest.java:114)
        at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 2319 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterDelete
   [junit4]   2> okt 12, 2018 12:16:03 AM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestIndexWriterDelete
   [junit4]   2>    1) Thread[id=10312, name=TEST-TestIndexWriterDelete.testUpdatesOnDiskFull-seed#[B468FA9DABA4598], state=TIMED_WAITING, group=TGRP-TestIndexWriterDelete]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.lucene.store.SlowOpeningMockIndexInputWrapper.<init>(SlowOpeningMockIndexInputWrapper.java:33)
   [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:769)
   [junit4]   2>         at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.<init>(Lucene50PostingsReader.java:81)
   [junit4]   2>         at org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:442)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372)
   [junit4]   2>         at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113)
   [junit4]   2>         at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
   [junit4]   2>         at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:678)
   [junit4]   2>         at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
   [junit4]   2>         at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:285)
   [junit4]   2>         at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:281)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterDelete.doTestOperationsOnDiskFull(TestIndexWriterDelete.java:638)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterDelete.testUpdatesOnDiskFull(TestIndexWriterDelete.java:490)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>    2) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    3) Thread[id=10311, name=SUITE-TestIndexWriterDelete-seed#[B468FA9DABA4598], state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:706)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:627)
   [junit4]   2>    4) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:637)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:494)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:269)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:394)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterDelete -Dtests.method=testUpdatesOnDiskFull -Dtests.seed=B468FA9DABA4598 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=nl-NL -Dtests.timezone=America/Rosario -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   7153s J1 | TestIndexWriterDelete.testUpdatesOnDiskFull <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([B468FA9DABA4598]:0)
   [junit4]   2> okt 12, 2018 12:16:03 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> okt 12, 2018 12:16:23 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.index.TestIndexWriterDelete:
   [junit4]   2>    1) Thread[id=10312, name=TEST-TestIndexWriterDelete.testUpdatesOnDiskFull-seed#[B468FA9DABA4598], state=TIMED_WAITING, group=TGRP-TestIndexWriterDelete]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.lucene.store.SlowOpeningMockIndexInputWrapper.<init>(SlowOpeningMockIndexInputWrapper.java:33)
   [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:769)
   [junit4]   2>         at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
   [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1069)
   [junit4]   2>         at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:91)
   [junit4]   2>         at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:361)
   [junit4]   2>         at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291)
   [junit4]   2>         at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:524)
   [junit4]   2>         at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
   [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:862)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterDelete.doTestOperationsOnDiskFull(TestIndexWriterDelete.java:699)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterDelete.testUpdatesOnDiskFull(TestIndexWriterDelete.java:490)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> okt 12, 2018 12:16:23 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=10312, name=TEST-TestIndexWriterDelete.testUpdatesOnDiskFull-seed#[B468FA9DABA4598], state=TIMED_WAITING, group=TGRP-TestIndexWriterDelete]
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterDelete_B468FA9DABA4598-001
   [junit4]   2> Oct 12, 2018 3:16:23 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneVarGapDocFreqInterval), body=PostingsFormat(name=LuceneFixedGap), value=Lucene50(blocksize=128), content=Lucene50(blocksize=128)}, docValues:{dv=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Lucene70), content=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=267, maxMBSortInHeap=6.411699527907805, sim=RandomSimilarity(queryNorm=false): {field=DFR I(ne)B2, id=DFR I(ne)B2, body=LM Jelinek-Mercer(0.100000), content=DFR I(n)B2}, locale=nl-NL, timezone=America/Rosario
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=35413016,total=445644800
   [junit4]   2> NOTE: All tests run in this JVM: [TestFlex, LimitedFiniteStringsIteratorTest, TestNearSpansOrdered, TestIntsRef, TestDocValuesScoring, TestReusableStringReader, TestGrowableByteArrayDataOutput, TestLucene50StoredFieldsFormatHighCompression, TestClassicSimilarity, TestLucene70NormsFormat, TestIndexWriterNRTIsCurrent, TestWordlistLoader, TestGraphTokenStreamFiniteStrings, TestFloatRangeFieldQueries, TestPersistentSnapshotDeletionPolicy, TestDemo, TestMmapDirectory, TestLongRangeFieldQueries, TestPolygon2D, TestIndexSorting, TestMatchNoDocsQuery, TestNoMergePolicy, TestNRTReaderCleanup, TestSimpleExplanationsWithFillerDocs, TestLevenshteinAutomata, TestTermQuery, TestSearch, TestFastDecompressionMode, TestBooleanScorer, TestIndexableField, TestBagOfPositions, TestDocumentWriter, TestNumericUtils, TestBlockPostingsFormat2, TestMultiCollector, TestRamUsageEstimator, TestDuelingCodecs, TestAllFilesHaveChecksumFooter, TestStressAdvance, TestRateLimiter, TestIndexWriterMaxDocs, TestTimeLimitingCollector, TestTryDelete, TestIntBlockPool, TestUpgradeIndexMergePolicy, TestTragicIndexWriterDeadlock, TestBasics, TestStressIndexing2, TestSoftDeletesDirectoryReaderWrapper, TestDocValuesIndexing, TestSpanFirstQuery, TestSloppyPhraseQuery2, TestSegmentTermEnum, TestSpanTermQuery, TestPrefixRandom, TestIndexWriterForceMerge, TestBytesRefHash, TestNRTCachingDirectory, TestIndexSearcher, TestPriorityQueue, TestIndexInput, TestPendingDeletes, TestTermVectors, TestFeatureField, TestSearchAfter, Test2BPagedBytes, TestFieldCacheRewriteMethod, TestPhraseQuery, TestMultiset, TestCharArrayMap, TestStringHelper, TestFixedLengthBytesRefArray, TestExitableDirectoryReader, TestSegmentCacheables, TestPositiveScoresOnlyCollector, TestNeverDelete, TestLatLonDocValuesQueries, TestRegexpRandom, TestGeoEncodingUtils, TestTimSorterWorstCase, TestCharFilter, TestCharacterUtils, TestDelegatingAnalyzerWrapper, TestBytesRefAttImpl, TestCharTermAttributeImpl, TestPackedTokenAttributeImpl, TestSimpleAttributeImpl, TestCodecUtil, TestForUtil, TestLucene50CompoundFormat, TestLucene50LiveDocsFormat, TestLucene60FieldInfoFormat, TestLucene60PointsFormat, TestIndexedDISI, TestPerFieldDocValuesFormat, TestBinaryDocument, TestDateTools, TestDocument, TestFieldType, TestIntRange, TestLatLonPoint, Test2BDocs, Test2BNumericDocValues, Test2BPoints, Test2BPositions, Test2BPostings, TestDirectoryReader, TestDirectoryReaderReopen, TestDoc, TestDocCount, TestDocIDMerger, TestDocInverterPerFieldErrorInfo, TestDocValues, TestFieldInvertState, TestFieldReuse, TestFieldsReader, TestFilterCodecReader, TestFilterDirectoryReader, TestFilterLeafReader, TestForTooMuchCloning, TestForceMergeForever, TestIndexCommit, TestIndexWriter, TestIndexWriterCommit, TestIndexWriterConfig, TestIndexWriterDelete]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterDelete -Dtests.seed=B468FA9DABA4598 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=nl-NL -Dtests.timezone=America/Rosario -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestIndexWriterDelete (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > at __randomizedtesting.SeedInfo.seed([B468FA9DABA4598]:0)
   [junit4] Completed [486/486 (1!)] on J1 in 7220.13s, 15 tests, 2 errors, 1 skipped <<< FAILURES!

[...truncated 11847 lines...]
   [junit4] Suite: org.apache.solr.cloud.MultiThreadedOCPTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/init-core-data-001
   [junit4]   2> 1782494 INFO  (SUITE-MultiThreadedOCPTest-seed#[951F32873E39A1C1]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1782496 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1782547 INFO  (Thread-3321) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1782547 INFO  (Thread-3321) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1782604 ERROR (Thread-3321) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1782640 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.ZkTestServer start zk server on port:45621
   [junit4]   2> 1782693 INFO  (zkConnectionManagerCallback-2215-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1783006 INFO  (zkConnectionManagerCallback-2217-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1783009 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1783023 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1783024 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1783025 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1783025 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1783043 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1783044 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1783044 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1783045 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1783046 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1783117 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1783206 INFO  (zkConnectionManagerCallback-2221-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1783315 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1784238 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1784239 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1784239 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1784239 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1784239 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@22710b88{/,null,AVAILABLE}
   [junit4]   2> 1784254 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4310dba7{SSL,[ssl, http/1.1]}{127.0.0.1:35753}
   [junit4]   2> 1784254 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.Server Started @1784693ms
   [junit4]   2> 1784254 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=35753, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/control-001/cores}
   [junit4]   2> 1784254 ERROR (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1784254 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1784254 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 1784254 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1784254 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1784254 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-12T07:31:57.583Z
   [junit4]   2> 1784289 INFO  (zkConnectionManagerCallback-2223-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1784310 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1784310 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/control-001/solr.xml
   [junit4]   2> 1784314 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1784314 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1784316 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1784718 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45621/solr
   [junit4]   2> 1784760 INFO  (zkConnectionManagerCallback-2227-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1784762 INFO  (zkConnectionManagerCallback-2229-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1785714 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [n:127.0.0.1:35753_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35753_
   [junit4]   2> 1785715 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [n:127.0.0.1:35753_    ] o.a.s.c.Overseer Overseer (id=73477557099495429-127.0.0.1:35753_-n_0000000000) starting
   [junit4]   2> 1785783 INFO  (zkConnectionManagerCallback-2236-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1785799 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [n:127.0.0.1:35753_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45621/solr ready
   [junit4]   2> 1785807 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [n:127.0.0.1:35753_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35753_
   [junit4]   2> 1785827 INFO  (OverseerStateUpdate-73477557099495429-127.0.0.1:35753_-n_0000000000) [n:127.0.0.1:35753_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:35753_
   [junit4]   2> 1785828 INFO  (zkCallback-2235-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1785829 INFO  (OverseerStateUpdate-73477557099495429-127.0.0.1:35753_-n_0000000000) [n:127.0.0.1:35753_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1785948 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [n:127.0.0.1:35753_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1786004 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [n:127.0.0.1:35753_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1786040 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [n:127.0.0.1:35753_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1786041 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [n:127.0.0.1:35753_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1786054 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [n:127.0.0.1:35753_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/control-001/cores
   [junit4]   2> 1786153 INFO  (zkConnectionManagerCallback-2242-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1786154 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1786165 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45621/solr ready
   [junit4]   2> 1786249 INFO  (qtp1499665219-10248) [n:127.0.0.1:35753_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35753_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1786309 INFO  (OverseerThreadFactory-4169-thread-1-processing-n:127.0.0.1:35753_) [n:127.0.0.1:35753_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1786430 INFO  (qtp1499665219-10244) [n:127.0.0.1:35753_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1786472 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1786472 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1787564 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1787724 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1788291 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1788415 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1788416 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1788416 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1788419 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=36.86328125, floorSegmentMB=1.0703125, forceMergeDeletesPctAllowed=1.4196028447457099, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=31.621062214717984
   [junit4]   2> 1788432 WARN  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1788673 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1788673 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1788675 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1788675 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1788676 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=23, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6404211228504807]
   [junit4]   2> 1788676 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@b4503d2[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1788701 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1788702 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1788703 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1788703 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1614103992894226432
   [junit4]   2> 1788704 INFO  (searcherExecutor-4174-thread-1-processing-n:127.0.0.1:35753_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@b4503d2[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1788756 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 1788772 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1788772 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1788772 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:35753/control_collection_shard1_replica_n1/
   [junit4]   2> 1788772 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1788772 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:35753/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1788772 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1788784 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:35753/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1788844 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1788863 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2391
   [junit4]   2> 1788887 INFO  (qtp1499665219-10248) [n:127.0.0.1:35753_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1788949 INFO  (zkCallback-2228-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1789887 INFO  (qtp1499665219-10248) [n:127.0.0.1:35753_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35753_&wt=javabin&version=2} status=0 QTime=3638
   [junit4]   2> 1789962 INFO  (zkConnectionManagerCallback-2247-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1789963 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1789963 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45621/solr ready
   [junit4]   2> 1789964 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1789982 INFO  (qtp1499665219-10251) [n:127.0.0.1:35753_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1789990 INFO  (OverseerThreadFactory-4169-thread-2-processing-n:127.0.0.1:35753_) [n:127.0.0.1:35753_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 1790015 INFO  (OverseerCollectionConfigSetProcessor-73477557099495429-127.0.0.1:35753_-n_0000000000) [n:127.0.0.1:35753_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1790116 WARN  (OverseerThreadFactory-4169-thread-2-processing-n:127.0.0.1:35753_) [n:127.0.0.1:35753_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 1790119 INFO  (qtp1499665219-10251) [n:127.0.0.1:35753_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1790119 INFO  (qtp1499665219-10251) [n:127.0.0.1:35753_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=137
   [junit4]   2> 1790698 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-1-001 of type NRT
   [junit4]   2> 1790699 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1790729 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1790729 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1790729 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1790729 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@b10005e{/,null,AVAILABLE}
   [junit4]   2> 1790730 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4fa070c9{SSL,[ssl, http/1.1]}{127.0.0.1:45158}
   [junit4]   2> 1790730 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.Server Started @1791169ms
   [junit4]   2> 1790730 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=45158, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-1-001/cores}
   [junit4]   2> 1790730 ERROR (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1790730 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1790730 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 1790730 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1790730 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1790730 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-12T07:32:04.059Z
   [junit4]   2> 1790732 INFO  (zkConnectionManagerCallback-2249-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1790745 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1790746 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-1-001/solr.xml
   [junit4]   2> 1790766 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1790766 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1790767 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1791466 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45621/solr
   [junit4]   2> 1791483 INFO  (zkConnectionManagerCallback-2253-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1791500 INFO  (zkConnectionManagerCallback-2255-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1791551 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1791581 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1791581 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45158_
   [junit4]   2> 1791583 INFO  (zkCallback-2235-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1791583 INFO  (zkCallback-2228-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1791610 INFO  (zkCallback-2254-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1791650 INFO  (zkCallback-2246-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1791682 INFO  (zkConnectionManagerCallback-2262-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1791683 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1791684 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45621/solr ready
   [junit4]   2> 1791685 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1791772 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1791811 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1791811 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1791825 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-1-001/cores
   [junit4]   2> 1791936 INFO  (qtp1499665219-10248) [n:127.0.0.1:35753_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:45158_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1791955 INFO  (OverseerCollectionConfigSetProcessor-73477557099495429-127.0.0.1:35753_-n_0000000000) [n:127.0.0.1:35753_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1791968 INFO  (qtp1499665219-10251) [n:127.0.0.1:35753_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1791970 INFO  (qtp1499665219-10249) [n:127.0.0.1:35753_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1792079 INFO  (qtp150942459-10302) [n:127.0.0.1:45158_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1792097 INFO  (OverseerThreadFactory-4169-thread-3-processing-n:127.0.0.1:35753_) [n:127.0.0.1:35753_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:45158_ for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 1792166 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_    x:collection1_shard2_replica_n41] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1793231 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1793300 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
   [junit4]   2> 1793795 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1793928 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
   [junit4]   2> 1793929 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1793929 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
   [junit4]   2> 1793948 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=36.86328125, floorSegmentMB=1.0703125, forceMergeDeletesPctAllowed=1.4196028447457099, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=31.621062214717984
   [junit4]   2> 1793953 WARN  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1794240 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1794240 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1794242 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1794242 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1794243 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=23, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6404211228504807]
   [junit4]   2> 1794244 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@4fc82158[collection1_shard2_replica_n41] main]
   [junit4]   2> 1794274 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1794293 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1794294 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1794294 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1614103998756814848
   [junit4]   2> 1794295 INFO  (searcherExecutor-4188-thread-1-processing-n:127.0.0.1:45158_ x:collection1_shard2_replica_n41 c:collection1 s:shard2) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@4fc82158[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1794314 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node42=0}, version=0}
   [junit4]   2> 1794316 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1794316 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1794316 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:45158/collection1_shard2_replica_n41/
   [junit4]   2> 1794316 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1794316 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy https://127.0.0.1:45158/collection1_shard2_replica_n41/ has no replicas
   [junit4]   2> 1794316 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1794333 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:45158/collection1_shard2_replica_n41/ shard2
   [junit4]   2> 1794384 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1794411 INFO  (qtp150942459-10298) [n:127.0.0.1:45158_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2245
   [junit4]   2> 1794414 INFO  (qtp1499665219-10248) [n:127.0.0.1:35753_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:45158_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=2477
   [junit4]   2> 1794500 INFO  (zkCallback-2254-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1795117 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-2-001 of type NRT
   [junit4]   2> 1795119 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1795130 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1795131 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1795131 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1795131 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@257fb3e0{/,null,AVAILABLE}
   [junit4]   2> 1795131 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d090374{SSL,[ssl, http/1.1]}{127.0.0.1:40269}
   [junit4]   2> 1795131 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.e.j.s.Server Started @1795571ms
   [junit4]   2> 1795131 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=40269, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-2-001/cores}
   [junit4]   2> 1795132 ERROR (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1795179 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1795179 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 1795179 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1795179 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1795179 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-12T07:32:08.508Z
   [junit4]   2> 1795181 INFO  (zkConnectionManagerCallback-2265-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1795188 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1795188 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-2-001/solr.xml
   [junit4]   2> 1795212 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1795212 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1795213 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1795840 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45621/solr
   [junit4]   2> 1795855 INFO  (zkConnectionManagerCallback-2269-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1795893 INFO  (zkConnectionManagerCallback-2271-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1795930 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1795950 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1795950 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40269_
   [junit4]   2> 1795954 INFO  (zkCallback-2254-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1795974 INFO  (zkCallback-2235-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1795982 INFO  (zkCallback-2228-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1795991 INFO  (zkCallback-2246-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1796011 INFO  (OverseerCollectionConfigSetProcessor-73477557099495429-127.0.0.1:35753_-n_0000000000) [n:127.0.0.1:35753_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1796018 INFO  (zkCallback-2270-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1796026 INFO  (zkCallback-2261-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1796082 INFO  (zkConnectionManagerCallback-2278-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1796100 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1796100 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45621/solr ready
   [junit4]   2> 1796148 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1796353 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1796444 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1796475 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1796476 INFO  (TEST-MultiThreadedOCPTest.test-seed#[951F32873E39A1C1]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-2-001/cores
   [junit4]   2> 1796670 INFO  (qtp1499665219-10251) [n:127.0.0.1:35753_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:40269_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1796779 INFO  (qtp1499665219-10248) [n:127.0.0.1:35753_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1796781 INFO  (qtp1499665219-10248) [n:127.0.0.1:35753_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1796975 INFO  (qtp271459861-10339) [n:127.0.0.1:40269_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1797011 INFO  (qtp150942459-10304) [n:127.0.0.1:45158_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard2.replica_n41:INDEX.sizeInBytes} status=0 QTime=1
   [junit4]   2> 1797012 INFO  (qtp150942459-10305) [n:127.0.0.1:45158_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1797034 INFO  (OverseerThreadFactory-4169-thread-4-processing-n:127.0.0.1:35753_) [n:127.0.0.1:35753_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:40269_ for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 1797154 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_    x:collection1_shard1_replica_n43] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1797195 INFO  (zkCallback-2254-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1797322 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1797414 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema [collection1_shard1_replica_n43] Schema name=test
   [junit4]   2> 1798581 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1798707 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n43' using configuration from collection collection1, trusted=true
   [junit4]   2> 1798707 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n43' (registry 'solr.core.collection1.shard1.replica_n43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2d3d5307
   [junit4]   2> 1798707 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [[collection1_shard1_replica_n43] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-2-001/cores/collection1_shard1_replica_n43], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.MultiThreadedOCPTest_951F32873E39A1C1-001/shard-2-001/cores/collection1_shard1_replica_n43/data/]
   [junit4]   2> 1798727 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=36.86328125, floorSegmentMB=1.0703125, forceMergeDeletesPctAllowed=1.4196028447457099, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=31.621062214717984
   [junit4]   2> 1798743 WARN  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1799128 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1799128 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1799129 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1799129 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1799145 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=23, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6404211228504807]
   [junit4]   2> 1799145 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.s.SolrIndexSearcher Opening [Searcher@72fb14dc[collection1_shard1_replica_n43] main]
   [junit4]   2> 1799161 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1799165 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1799166 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1799166 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1614104003865477120
   [junit4]   2> 1799202 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node44=0}, version=0}
   [junit4]   2> 1799204 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1799204 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1799204 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:40269/collection1_shard1_replica_n43/
   [junit4]   2> 1799204 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1799204 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy https://127.0.0.1:40269/collection1_shard1_replica_n43/ has no replicas
   [junit4]   2> 1799204 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1799238 INFO  (searcherExecutor-4202-thread-1-processing-n:127.0.0.1:40269_ x:collection1_shard1_replica_n43 c:collection1 s:shard1) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [collection1_shard1_replica_n43] Registered new searcher Searcher@72fb14dc[collection1_shard1_replica_n43] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1799293 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:40269/collection1_shard1_replica_n43/ shard1
   [junit4]   2> 1799293 INFO  (zkCallback-2254-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1799321 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collection1 s:shard1  x:collection1_shard1_replica_n43] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1799372 INFO  (qtp271459861-10335) [n:127.0.0.1:40269_ c:collecti

[...truncated too long message...]

.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1357551203
     [copy] Copying 239 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1357551203
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/null1357551203

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 37ms :: artifacts dl 11ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 646 minutes 41 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]