[JENKINS] Lucene-Solr-Tests-7.x - Build # 934 - Unstable

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 934 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/934/

7 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore

Error Message:
ObjectTracker found 7 object(s) that were not released!!! [InternalHttpClient, SolrCore, MMapDirectory, MMapDirectory, InternalHttpClient, MMapDirectory, MMapDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:226)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:268)  at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1187)  at org.apache.solr.cloud.ReplicateFromLeader.startReplication(ReplicateFromLeader.java:106)  at org.apache.solr.cloud.ZkController.startReplicationFromLeader(ZkController.java:1238)  at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:381)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:312)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:294)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1045)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:869)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1138)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:768)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:960)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:869)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1138)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:504)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:347)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:421)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1156)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:226)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:268)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:417)  at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:227)  at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:363)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:312)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:294)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:358)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:737)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:960)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:869)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1138)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:507)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:954)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:869)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1138)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 7 object(s) that were not released!!! [InternalHttpClient, SolrCore, MMapDirectory, MMapDirectory, InternalHttpClient, MMapDirectory, MMapDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
        at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:226)
        at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:268)
        at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1187)
        at org.apache.solr.cloud.ReplicateFromLeader.startReplication(ReplicateFromLeader.java:106)
        at org.apache.solr.cloud.ZkController.startReplicationFromLeader(ZkController.java:1238)
        at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:381)
        at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:312)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:294)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1045)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:869)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1138)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:768)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:960)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:869)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1138)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:504)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:347)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:421)
        at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1156)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
        at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:226)
        at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:268)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:417)
        at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:227)
        at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:363)
        at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:312)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:294)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:358)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:737)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:960)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:869)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1138)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:507)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:954)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:869)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1138)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1048)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([27274EB8FF9FC467]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:304)
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds

Error Message:
did not finish processing in time

Stack Trace:
java.lang.AssertionError: did not finish processing in time
        at __randomizedtesting.SeedInfo.seed([27274EB8FF9FC467:2DA4F115B224CF3D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds(IndexSizeTriggerTest.java:572)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestSimLargeCluster.testNodeLost

Error Message:
last ClusterState: znodeVersion: 236 live nodes:[127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10052_solr, 127.0.0.1:10079_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10009_solr, 127.0.0.1:10098_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10071_solr, 127.0.0.1:10063_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10007_solr, 127.0.0.1:10015_solr, 127.0.0.1:10055_solr, 127.0.0.1:10069_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10062_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10008_solr, 127.0.0.1:10024_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10045_solr, 127.0.0.1:10100_solr, 127.0.0.1:10042_solr, 127.0.0.1:10037_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10103_solr, 127.0.0.1:10064_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr] collections:{.system=DocCollection(.system//clusterstate.json/235)={   "replicationFactor":"3",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{"shard1":{       "replicas":{         "core_node1":{           "core":".system_shard1_replica_n1",           "leader":"true",           "SEARCHER.searcher.maxDoc":132,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":76240,           "node_name":"127.0.0.1:10014_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":7.100403308868408E-5,           "SEARCHER.searcher.numDocs":132},         "core_node2":{           "core":".system_shard1_replica_n2",           "SEARCHER.searcher.maxDoc":132,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":76240,           "node_name":"127.0.0.1:10039_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":7.100403308868408E-5,           "SEARCHER.searcher.numDocs":132},         "core_node3":{           "core":".system_shard1_replica_n3",           "SEARCHER.searcher.maxDoc":132,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":76240,           "node_name":"127.0.0.1:10047_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":7.100403308868408E-5,           "SEARCHER.searcher.numDocs":132}},       "range":"80000000-7fffffff",       "state":"active"}}}, testNodeLost=DocCollection(testNodeLost//clusterstate.json/235)={   "replicationFactor":"1",   "pullReplicas":"0",   "router":{"name":"compositeId"},   "maxShardsPerNode":"5",   "autoAddReplicas":"false",   "nrtReplicas":"10",   "tlogReplicas":"0",   "autoCreated":"true",   "shards":{     "shard20":{       "replicas":{         "core_node191":{           "core":"testNodeLost_shard20_replica_n191",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10011_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node206":{           "node_name":"127.0.0.1:10103_solr",           "core":"testNodeLost_shard20_replica_n205",           "state":"active",           "INDEX.sizeInBytes":10240,           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6},         "core_node200":{           "core":"testNodeLost_shard20_replica_n200",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10061_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node199":{           "core":"testNodeLost_shard20_replica_n199",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10037_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node198":{           "core":"testNodeLost_shard20_replica_n198",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10077_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node196":{           "core":"testNodeLost_shard20_replica_n196",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10050_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node195":{           "core":"testNodeLost_shard20_replica_n195",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10028_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node194":{           "core":"testNodeLost_shard20_replica_n194",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10078_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node193":{           "core":"testNodeLost_shard20_replica_n193",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10105_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node192":{           "core":"testNodeLost_shard20_replica_n192",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10081_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"73330000-7fffffff",       "state":"active"},     "shard10":{       "replicas":{         "core_node92":{           "core":"testNodeLost_shard10_replica_n92",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10081_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node91":{           "core":"testNodeLost_shard10_replica_n91",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10011_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node94":{           "core":"testNodeLost_shard10_replica_n94",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10078_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node93":{           "core":"testNodeLost_shard10_replica_n93",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10105_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node96":{           "core":"testNodeLost_shard10_replica_n96",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10050_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node95":{           "core":"testNodeLost_shard10_replica_n95",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10028_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node98":{           "core":"testNodeLost_shard10_replica_n98",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10077_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node100":{           "core":"testNodeLost_shard10_replica_n100",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10061_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node99":{           "core":"testNodeLost_shard10_replica_n99",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10037_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node208":{           "node_name":"127.0.0.1:10059_solr",           "core":"testNodeLost_shard10_replica_n207",           "state":"active",           "INDEX.sizeInBytes":10240,           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6}},       "range":"f3330000-ffffffff",       "state":"active"},     "shard11":{       "replicas":{         "core_node108":{           "core":"testNodeLost_shard11_replica_n108",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10084_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node107":{           "core":"testNodeLost_shard11_replica_n107",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10093_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node106":{           "core":"testNodeLost_shard11_replica_n106",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10098_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node105":{           "core":"testNodeLost_shard11_replica_n105",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10071_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node104":{           "core":"testNodeLost_shard11_replica_n104",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10072_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node103":{           "core":"testNodeLost_shard11_replica_n103",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10103_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node102":{           "core":"testNodeLost_shard11_replica_n102",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10099_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node101":{           "core":"testNodeLost_shard11_replica_n101",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10013_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node110":{           "core":"testNodeLost_shard11_replica_n110",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10085_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node109":{           "core":"testNodeLost_shard11_replica_n109",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10024_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"0-ccbffff",       "state":"active"},     "shard12":{       "replicas":{         "core_node119":{           "core":"testNodeLost_shard12_replica_n119",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10069_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node118":{           "core":"testNodeLost_shard12_replica_n118",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10054_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node117":{           "core":"testNodeLost_shard12_replica_n117",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10029_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node116":{           "core":"testNodeLost_shard12_replica_n116",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10044_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node115":{           "core":"testNodeLost_shard12_replica_n115",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10094_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node114":{           "core":"testNodeLost_shard12_replica_n114",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10091_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node113":{           "core":"testNodeLost_shard12_replica_n113",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10048_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node112":{           "core":"testNodeLost_shard12_replica_n112",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10075_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node111":{           "core":"testNodeLost_shard12_replica_n111",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10046_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node120":{           "core":"testNodeLost_shard12_replica_n120",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10036_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"ccc0000-1998ffff",       "state":"active"},     "shard2":{       "replicas":{         "core_node18":{           "core":"testNodeLost_shard2_replica_n18",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10054_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node17":{           "core":"testNodeLost_shard2_replica_n17",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10029_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node19":{           "core":"testNodeLost_shard2_replica_n19",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10069_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node20":{           "core":"testNodeLost_shard2_replica_n20",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10036_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node12":{           "core":"testNodeLost_shard2_replica_n12",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10075_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node11":{           "core":"testNodeLost_shard2_replica_n11",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10046_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node14":{           "core":"testNodeLost_shard2_replica_n14",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10091_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node13":{           "core":"testNodeLost_shard2_replica_n13",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10048_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node16":{           "core":"testNodeLost_shard2_replica_n16",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10044_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node15":{           "core":"testNodeLost_shard2_replica_n15",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10094_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"8ccc0000-9998ffff",       "state":"active"},     "shard13":{       "replicas":{         "core_node129":{           "core":"testNodeLost_shard13_replica_n129",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10023_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node128":{           "core":"testNodeLost_shard13_replica_n128",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10034_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node127":{           "core":"testNodeLost_shard13_replica_n127",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10083_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node126":{           "core":"testNodeLost_shard13_replica_n126",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10021_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node125":{           "core":"testNodeLost_shard13_replica_n125",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10020_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node124":{           "core":"testNodeLost_shard13_replica_n124",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10104_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node123":{           "core":"testNodeLost_shard13_replica_n123",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10019_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node122":{           "core":"testNodeLost_shard13_replica_n122",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10076_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node121":{           "core":"testNodeLost_shard13_replica_n121",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10030_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node130":{           "core":"testNodeLost_shard13_replica_n130",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10014_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"19990000-2665ffff",       "state":"active"},     "shard14":{       "replicas":{         "core_node139":{           "core":"testNodeLost_shard14_replica_n139",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10073_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node138":{           "core":"testNodeLost_shard14_replica_n138",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10045_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node137":{           "core":"testNodeLost_shard14_replica_n137",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10009_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node136":{           "core":"testNodeLost_shard14_replica_n136",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10041_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node135":{           "core":"testNodeLost_shard14_replica_n135",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10022_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node134":{           "core":"testNodeLost_shard14_replica_n134",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10102_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node133":{           "core":"testNodeLost_shard14_replica_n133",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10065_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node132":{           "core":"testNodeLost_shard14_replica_n132",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10025_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node131":{           "core":"testNodeLost_shard14_replica_n131",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10059_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node140":{           "core":"testNodeLost_shard14_replica_n140",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10035_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"26660000-3332ffff",       "state":"active"},     "shard3":{       "replicas":{         "core_node29":{           "core":"testNodeLost_shard3_replica_n29",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10023_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node28":{           "core":"testNodeLost_shard3_replica_n28",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10034_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node30":{           "core":"testNodeLost_shard3_replica_n30",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10014_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node21":{           "core":"testNodeLost_shard3_replica_n21",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10030_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node23":{           "core":"testNodeLost_shard3_replica_n23",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10019_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node22":{           "core":"testNodeLost_shard3_replica_n22",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10076_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node25":{           "core":"testNodeLost_shard3_replica_n25",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10020_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node24":{           "core":"testNodeLost_shard3_replica_n24",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10104_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node27":{           "core":"testNodeLost_shard3_replica_n27",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10083_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node26":{           "core":"testNodeLost_shard3_replica_n26",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10021_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"99990000-a665ffff",       "state":"active"},     "shard15":{       "replicas":{         "core_node149":{           "core":"testNodeLost_shard15_replica_n149",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10042_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node148":{           "core":"testNodeLost_shard15_replica_n148",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10043_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node147":{           "core":"testNodeLost_shard15_replica_n147",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10106_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node146":{           "core":"testNodeLost_shard15_replica_n146",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10067_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node145":{           "core":"testNodeLost_shard15_replica_n145",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10026_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node144":{           "core":"testNodeLost_shard15_replica_n144",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10089_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node143":{           "core":"testNodeLost_shard15_replica_n143",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10092_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node142":{           "core":"testNodeLost_shard15_replica_n142",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10074_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node141":{           "core":"testNodeLost_shard15_replica_n141",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10047_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node150":{           "core":"testNodeLost_shard15_replica_n150",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10068_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"33330000-3fffffff",       "state":"active"},     "shard4":{       "replicas":{         "core_node39":{           "core":"testNodeLost_shard4_replica_n39",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10073_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node40":{           "core":"testNodeLost_shard4_replica_n40",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10035_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node32":{           "core":"testNodeLost_shard4_replica_n32",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10025_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node31":{           "core":"testNodeLost_shard4_replica_n31",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10059_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node34":{           "core":"testNodeLost_shard4_replica_n34",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10102_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node33":{           "core":"testNodeLost_shard4_replica_n33",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10065_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node36":{           "core":"testNodeLost_shard4_replica_n36",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10041_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node35":{           "core":"testNodeLost_shard4_replica_n35",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10022_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node38":{           "core":"testNodeLost_shard4_replica_n38",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10045_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node37":{           "core":"testNodeLost_shard4_replica_n37",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10009_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"a6660000-b332ffff",       "state":"active"},     "shard5":{       "replicas":{         "core_node50":{           "core":"testNodeLost_shard5_replica_n50",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10068_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node41":{           "core":"testNodeLost_shard5_replica_n41",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10047_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node43":{           "core":"testNodeLost_shard5_replica_n43",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10092_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node42":{           "core":"testNodeLost_shard5_replica_n42",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10074_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node45":{           "core":"testNodeLost_shard5_replica_n45",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10026_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node44":{           "core":"testNodeLost_shard5_replica_n44",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10089_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node47":{           "core":"testNodeLost_shard5_replica_n47",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10106_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node46":{           "core":"testNodeLost_shard5_replica_n46",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10067_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node49":{           "core":"testNodeLost_shard5_replica_n49",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10042_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node48":{           "core":"testNodeLost_shard5_replica_n48",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10043_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"b3330000-bfffffff",       "state":"active"},     "shard16":{       "replicas":{         "core_node159":{           "core":"testNodeLost_shard16_replica_n159",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10008_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node158":{           "core":"testNodeLost_shard16_replica_n158",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10040_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node157":{           "core":"testNodeLost_shard16_replica_n157",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10080_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node156":{           "core":"testNodeLost_shard16_replica_n156",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10057_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node155":{           "core":"testNodeLost_shard16_replica_n155",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10095_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node154":{           "core":"testNodeLost_shard16_replica_n154",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10066_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node153":{           "core":"testNodeLost_shard16_replica_n153",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10063_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node152":{           "core":"testNodeLost_shard16_replica_n152",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10056_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node151":{           "core":"testNodeLost_shard16_replica_n151",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10079_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node160":{           "core":"testNodeLost_shard16_replica_n160",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10051_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"40000000-4ccbffff",       "state":"active"},     "shard17":{       "replicas":{         "core_node169":{           "core":"testNodeLost_shard17_replica_n169",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10031_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node202":{           "node_name":"127.0.0.1:10072_solr",           "core":"testNodeLost_shard17_replica_n201",           "state":"active",           "INDEX.sizeInBytes":10240,           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6},         "core_node168":{           "core":"testNodeLost_shard17_replica_n168",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10027_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node167":{           "core":"testNodeLost_shard17_replica_n167",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10015_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node166":{           "core":"testNodeLost_shard17_replica_n166",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10064_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node165":{           "core":"testNodeLost_shard17_replica_n165",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10058_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node164":{           "core":"testNodeLost_shard17_replica_n164",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10038_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node162":{           "core":"testNodeLost_shard17_replica_n162",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10017_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node161":{           "core":"testNodeLost_shard17_replica_n161",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10049_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node170":{           "core":"testNodeLost_shard17_replica_n170",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10097_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"4ccc0000-5998ffff",       "state":"active"},     "shard18":{       "replicas":{         "core_node180":{           "core":"testNodeLost_shard18_replica_n180",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10052_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node179":{           "core":"testNodeLost_shard18_replica_n179",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10086_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node178":{           "core":"testNodeLost_shard18_replica_n178",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10010_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node177":{           "core":"testNodeLost_shard18_replica_n177",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10012_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node176":{           "core":"testNodeLost_shard18_replica_n176",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10060_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node175":{           "core":"testNodeLost_shard18_replica_n175",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10087_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node174":{           "core":"testNodeLost_shard18_replica_n174",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10032_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node173":{           "core":"testNodeLost_shard18_replica_n173",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10096_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node172":{           "core":"testNodeLost_shard18_replica_n172",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10070_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node171":{           "core":"testNodeLost_shard18_replica_n171",           "leader":"true",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10053_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0}},       "range":"59990000-6665ffff",       "state":"active"},     "shard19":{       "replicas":{         "core_node190":{           "core":"testNodeLost_shard19_replica_n190",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10016_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node189":{           "core":"testNodeLost_shard19_replica_n189",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10039_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node188":{           "core":"testNodeLost_shard19_replica_n188",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10033_solr",           "state":"down",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node187":{           "core":"testNodeLost_shard19_replica_n187",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10088_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node186":{           "core":"testNodeLost_shard19_replica_n186",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10062_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node185":{           "core":"testNodeLost_shard19_replica_n185",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10007_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node184":{           "core":"testNodeLost_shard19_replica_n184",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10100_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node183":{           "core":"testNodeLost_shard19_replica_n183",           "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":10240,           "node_name":"127.0.0.1:10101_solr",           "state":"active",           "type":"NRT",           "INDEX.sizeInGB":9.5367431640625E-6,           "SEARCHER.searcher.numDocs":0},         "core_node182":{           "core":"testNodeLost_shard19_replic

[...truncated too long message...]

rty disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/null2042831335
     [copy] Copying 239 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/null2042831335
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/null2042831335

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 74ms :: artifacts dl 1ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 208 minutes 24 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 935 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/935/

6 tests failed.
FAILED:  org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes

Error Message:
[127.0.0.1:40882_solr] expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: [127.0.0.1:40882_solr] expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([2102D9FBAE552B33:45420C97159BA322]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes(CollectionsAPIDistributedZkTest.java:351)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:36731_solr, 127.0.0.1:46622_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/18)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node5":{           "core":"testSimple1_shard1_replica_n2",           "base_url":"http://127.0.0.1:36731/solr",           "node_name":"127.0.0.1:36731_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node12":{           "core":"testSimple1_shard1_replica_n11",           "base_url":"http://127.0.0.1:36731/solr",           "node_name":"127.0.0.1:36731_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "core":"testSimple1_shard2_replica_n4",           "base_url":"http://127.0.0.1:36731/solr",           "node_name":"127.0.0.1:36731_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node10":{           "core":"testSimple1_shard2_replica_n9",           "base_url":"http://127.0.0.1:43955/solr",           "node_name":"127.0.0.1:43955_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:36731_solr, 127.0.0.1:46622_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/18)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node5":{
          "core":"testSimple1_shard1_replica_n2",
          "base_url":"http://127.0.0.1:36731/solr",
          "node_name":"127.0.0.1:36731_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"http://127.0.0.1:36731/solr",
          "node_name":"127.0.0.1:36731_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "core":"testSimple1_shard2_replica_n4",
          "base_url":"http://127.0.0.1:36731/solr",
          "node_name":"127.0.0.1:36731_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"http://127.0.0.1:43955/solr",
          "node_name":"127.0.0.1:43955_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([2102D9FBAE552B33:19B1FD0589A6FFE2]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:33422_solr, 127.0.0.1:38004_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/16)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "dataDir":"hdfs://localhost:32931/data/testSimple1/core_node3/data/",           "base_url":"http://127.0.0.1:38004/solr",           "node_name":"127.0.0.1:38004_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost:32931/data/testSimple1/core_node3/data/tlog",           "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node4":{           "dataDir":"hdfs://localhost:32931/data/testSimple1/core_node4/data/",           "base_url":"http://127.0.0.1:38004/solr",           "node_name":"127.0.0.1:38004_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost:32931/data/testSimple1/core_node4/data/tlog",           "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",           "state":"active"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "dataDir":"hdfs://localhost:32931/data/testSimple1/core_node7/data/",           "base_url":"http://127.0.0.1:40251/solr",           "node_name":"127.0.0.1:40251_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost:32931/data/testSimple1/core_node7/data/tlog",           "core":"testSimple1_shard2_replica_n5",           "shared_storage":"true",           "state":"down"},         "core_node8":{           "dataDir":"hdfs://localhost:32931/data/testSimple1/core_node8/data/",           "base_url":"http://127.0.0.1:38004/solr",           "node_name":"127.0.0.1:38004_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost:32931/data/testSimple1/core_node8/data/tlog",           "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",           "state":"active",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:33422_solr, 127.0.0.1:38004_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/16)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://localhost:32931/data/testSimple1/core_node3/data/",
          "base_url":"http://127.0.0.1:38004/solr",
          "node_name":"127.0.0.1:38004_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost:32931/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node4":{
          "dataDir":"hdfs://localhost:32931/data/testSimple1/core_node4/data/",
          "base_url":"http://127.0.0.1:38004/solr",
          "node_name":"127.0.0.1:38004_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost:32931/data/testSimple1/core_node4/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"active"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://localhost:32931/data/testSimple1/core_node7/data/",
          "base_url":"http://127.0.0.1:40251/solr",
          "node_name":"127.0.0.1:40251_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost:32931/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n5",
          "shared_storage":"true",
          "state":"down"},
        "core_node8":{
          "dataDir":"hdfs://localhost:32931/data/testSimple1/core_node8/data/",
          "base_url":"http://127.0.0.1:38004/solr",
          "node_name":"127.0.0.1:38004_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost:32931/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([2102D9FBAE552B33:19B1FD0589A6FFE2]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew

Error Message:
expected:<200> but was:<403>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<403>
        at __randomizedtesting.SeedInfo.seed([2102D9FBAE552B33:16992DE59699F697]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.renewDelegationToken(TestDelegationWithHadoopAuth.java:120)
        at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.verifyDelegationTokenRenew(TestDelegationWithHadoopAuth.java:302)
        at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew(TestDelegationWithHadoopAuth.java:320)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest

Error Message:
Response was not received from shards on a single node

Stack Trace:
java.lang.AssertionError: Response was not received from shards on a single node
        at __randomizedtesting.SeedInfo.seed([857D0EF42B9A0198:79B0964DD54D9662]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.queryWithShardsPreferenceRules(CloudSolrClientTest.java:471)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest(CloudSolrClientTest.java:421)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([857D0EF42B9A0198:2D93C967D5D37594]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests(CloudSolrClientTest.java:550)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13349 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest
   [junit4]   2> 1280577 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/init-core-data-001
   [junit4]   2> 1280578 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=7 numCloses=7
   [junit4]   2> 1280579 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1280581 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1280582 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-001
   [junit4]   2> 1280582 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1280582 INFO  (Thread-2130) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1280582 INFO  (Thread-2130) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1280587 ERROR (Thread-2130) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1280682 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:34801
   [junit4]   2> 1280685 INFO  (zkConnectionManagerCallback-3010-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1280688 INFO  (jetty-launcher-3007-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1280688 INFO  (jetty-launcher-3007-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1280689 INFO  (jetty-launcher-3007-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1280689 INFO  (jetty-launcher-3007-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1280689 INFO  (jetty-launcher-3007-thread-2) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1280690 INFO  (jetty-launcher-3007-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@29145186{/solr,null,AVAILABLE}
   [junit4]   2> 1280691 INFO  (jetty-launcher-3007-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3e88b83b{HTTP/1.1,[http/1.1]}{127.0.0.1:33422}
   [junit4]   2> 1280691 INFO  (jetty-launcher-3007-thread-2) [    ] o.e.j.s.Server Started @1280745ms
   [junit4]   2> 1280691 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33422}
   [junit4]   2> 1280691 ERROR (jetty-launcher-3007-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1280692 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1280692 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 1280692 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1280692 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1280692 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T14:25:04.176Z
   [junit4]   2> 1280694 INFO  (jetty-launcher-3007-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1280694 INFO  (jetty-launcher-3007-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1280694 INFO  (jetty-launcher-3007-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1280695 INFO  (jetty-launcher-3007-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@53d9d090{/solr,null,AVAILABLE}
   [junit4]   2> 1280695 INFO  (jetty-launcher-3007-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@784dc103{HTTP/1.1,[http/1.1]}{127.0.0.1:38004}
   [junit4]   2> 1280695 INFO  (jetty-launcher-3007-thread-1) [    ] o.e.j.s.Server Started @1280749ms
   [junit4]   2> 1280695 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38004}
   [junit4]   2> 1280695 ERROR (jetty-launcher-3007-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1280696 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1280696 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 1280696 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1280696 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1280696 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T14:25:04.180Z
   [junit4]   2> 1280697 INFO  (zkConnectionManagerCallback-3014-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1280698 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1280702 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1280702 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1280709 INFO  (jetty-launcher-3007-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1280710 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1280711 INFO  (jetty-launcher-3007-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1280711 INFO  (jetty-launcher-3007-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1280711 INFO  (jetty-launcher-3007-thread-3) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1280711 INFO  (jetty-launcher-3007-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1086dca5{/solr,null,AVAILABLE}
   [junit4]   2> 1280712 INFO  (jetty-launcher-3007-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6ebe0fc2{HTTP/1.1,[http/1.1]}{127.0.0.1:40251}
   [junit4]   2> 1280712 INFO  (jetty-launcher-3007-thread-3) [    ] o.e.j.s.Server Started @1280766ms
   [junit4]   2> 1280712 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40251}
   [junit4]   2> 1280712 ERROR (jetty-launcher-3007-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1280712 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1280712 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.6.0
   [junit4]   2> 1280712 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1280712 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1280712 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T14:25:04.196Z
   [junit4]   2> 1280714 INFO  (zkConnectionManagerCallback-3012-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1280715 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1280718 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1280718 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1280720 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1280727 INFO  (zkConnectionManagerCallback-3016-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1280728 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1280731 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1280731 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1280733 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1280832 INFO  (jetty-launcher-3007-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34801/solr
   [junit4]   2> 1280837 INFO  (zkConnectionManagerCallback-3020-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1280846 INFO  (zkConnectionManagerCallback-3022-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1280874 INFO  (jetty-launcher-3007-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34801/solr
   [junit4]   2> 1280881 INFO  (zkConnectionManagerCallback-3028-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1280901 INFO  (zkConnectionManagerCallback-3030-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1280915 INFO  (jetty-launcher-3007-thread-2) [n:127.0.0.1:33422_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33422_solr
   [junit4]   2> 1280917 INFO  (jetty-launcher-3007-thread-2) [n:127.0.0.1:33422_solr    ] o.a.s.c.Overseer Overseer (id=72072660294500359-127.0.0.1:33422_solr-n_0000000000) starting
   [junit4]   2> 1280928 INFO  (zkConnectionManagerCallback-3037-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1280935 INFO  (jetty-launcher-3007-thread-2) [n:127.0.0.1:33422_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34801/solr ready
   [junit4]   2> 1280938 INFO  (OverseerStateUpdate-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [n:127.0.0.1:33422_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:33422_solr
   [junit4]   2> 1280952 INFO  (jetty-launcher-3007-thread-2) [n:127.0.0.1:33422_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33422_solr
   [junit4]   2> 1280957 DEBUG (OverseerAutoScalingTriggerThread-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 1280958 DEBUG (OverseerAutoScalingTriggerThread-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1280959 DEBUG (OverseerAutoScalingTriggerThread-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 1280959 DEBUG (OverseerAutoScalingTriggerThread-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1280962 INFO  (zkCallback-3036-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1280962 INFO  (zkCallback-3029-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1280968 DEBUG (OverseerAutoScalingTriggerThread-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:33422_solr]
   [junit4]   2> 1280969 DEBUG (OverseerAutoScalingTriggerThread-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1280969 DEBUG (OverseerAutoScalingTriggerThread-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1280970 DEBUG (ScheduledTrigger-4992-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 1280997 INFO  (jetty-launcher-3007-thread-2) [n:127.0.0.1:33422_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1281021 INFO  (jetty-launcher-3007-thread-2) [n:127.0.0.1:33422_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1281028 INFO  (jetty-launcher-3007-thread-1) [n:127.0.0.1:38004_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1281033 INFO  (jetty-launcher-3007-thread-2) [n:127.0.0.1:33422_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1281034 INFO  (jetty-launcher-3007-thread-2) [n:127.0.0.1:33422_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1281034 INFO  (jetty-launcher-3007-thread-1) [n:127.0.0.1:38004_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1281034 INFO  (jetty-launcher-3007-thread-1) [n:127.0.0.1:38004_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38004_solr
   [junit4]   2> 1281035 INFO  (jetty-launcher-3007-thread-2) [n:127.0.0.1:33422_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-001/node2/.
   [junit4]   2> 1281036 INFO  (zkCallback-3029-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1281036 INFO  (zkCallback-3021-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1281037 INFO  (jetty-launcher-3007-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34801/solr
   [junit4]   2> 1281049 INFO  (zkCallback-3036-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1281051 INFO  (zkConnectionManagerCallback-3042-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1281056 INFO  (zkConnectionManagerCallback-3044-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1281061 INFO  (zkConnectionManagerCallback-3049-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1281062 INFO  (jetty-launcher-3007-thread-1) [n:127.0.0.1:38004_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1281064 INFO  (jetty-launcher-3007-thread-1) [n:127.0.0.1:38004_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34801/solr ready
   [junit4]   2> 1281065 INFO  (jetty-launcher-3007-thread-1) [n:127.0.0.1:38004_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1281065 INFO  (jetty-launcher-3007-thread-3) [n:127.0.0.1:40251_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1281071 INFO  (jetty-launcher-3007-thread-3) [n:127.0.0.1:40251_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1281071 INFO  (jetty-launcher-3007-thread-3) [n:127.0.0.1:40251_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40251_solr
   [junit4]   2> 1281072 INFO  (zkCallback-3021-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1281073 INFO  (zkCallback-3036-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1281073 INFO  (zkCallback-3048-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1281073 INFO  (zkCallback-3029-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1281074 INFO  (zkCallback-3043-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1281090 INFO  (jetty-launcher-3007-thread-1) [n:127.0.0.1:38004_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1281106 INFO  (jetty-launcher-3007-thread-1) [n:127.0.0.1:38004_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1281106 INFO  (zkConnectionManagerCallback-3057-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1281106 INFO  (jetty-launcher-3007-thread-1) [n:127.0.0.1:38004_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1281107 INFO  (jetty-launcher-3007-thread-3) [n:127.0.0.1:40251_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1281108 INFO  (jetty-launcher-3007-thread-1) [n:127.0.0.1:38004_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-001/node1/.
   [junit4]   2> 1281108 INFO  (jetty-launcher-3007-thread-3) [n:127.0.0.1:40251_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34801/solr ready
   [junit4]   2> 1281109 INFO  (jetty-launcher-3007-thread-3) [n:127.0.0.1:40251_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1281145 INFO  (jetty-launcher-3007-thread-3) [n:127.0.0.1:40251_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1281160 INFO  (jetty-launcher-3007-thread-3) [n:127.0.0.1:40251_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1281160 INFO  (jetty-launcher-3007-thread-3) [n:127.0.0.1:40251_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1281162 INFO  (jetty-launcher-3007-thread-3) [n:127.0.0.1:40251_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-001/node3/.
   [junit4]   2> 1281205 INFO  (zkConnectionManagerCallback-3060-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1281209 INFO  (zkConnectionManagerCallback-3065-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1281211 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1281212 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34801/solr ready
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1281269 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1281282 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1281285 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1281304 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_33653_hdfs____.fbxzz6/webapp
   [junit4]   2> 1281740 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33653
   [junit4]   2> 1281824 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1281826 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1281847 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_33386_datanode____f94nlk/webapp
   [junit4]   2> 1281972 DEBUG (ScheduledTrigger-4992-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1282266 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33386
   [junit4]   2> 1282369 WARN  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1282371 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1282404 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_35226_datanode____nlmszf/webapp
   [junit4]   2> 1282475 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-002/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:32931) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 1282513 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xd1168b267c7f: from storage DS-2bb61c32-fbb2-4fe7-ba65-25815522d518 node DatanodeRegistration(127.0.0.1:46596, datanodeUuid=fca8fd8b-e37d-4a96-83d4-997a3139e2e1, infoPort=43876, infoSecurePort=0, ipcPort=34844, storageInfo=lv=-56;cid=testClusterID;nsid=35051983;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 1282513 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xd1168b267c7f: from storage DS-ed3ac862-87c1-453d-b247-a94c618c1eb8 node DatanodeRegistration(127.0.0.1:46596, datanodeUuid=fca8fd8b-e37d-4a96-83d4-997a3139e2e1, infoPort=43876, infoSecurePort=0, ipcPort=34844, storageInfo=lv=-56;cid=testClusterID;nsid=35051983;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1282889 INFO  (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[2102D9FBAE552B33]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35226
   [junit4]   2> 1282973 DEBUG (ScheduledTrigger-4992-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1283223 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-002/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:32931) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 1283232 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xd116b65c4c03: from storage DS-8fcc56f1-43aa-49c8-8f4d-590db7a34d2e node DatanodeRegistration(127.0.0.1:39496, datanodeUuid=f5d5f6f3-fd81-4abe-be09-c5b7a33c29c2, infoPort=37162, infoSecurePort=0, ipcPort=41038, storageInfo=lv=-56;cid=testClusterID;nsid=35051983;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 1283232 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0xd116b65c4c03: from storage DS-439b6082-eedc-4d5e-904a-79112a7c7115 node DatanodeRegistration(127.0.0.1:39496, datanodeUuid=f5d5f6f3-fd81-4abe-be09-c5b7a33c29c2, infoPort=37162, infoSecurePort=0, ipcPort=41038, storageInfo=lv=-56;cid=testClusterID;nsid=35051983;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1283630 INFO  (TEST-HdfsAutoAddReplicasIntegrationTest.testSimple-seed#[2102D9FBAE552B33]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSimple
   [junit4]   2> 1283645 INFO  (qtp1006018741-11223) [n:127.0.0.1:33422_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=true&name=testSimple1&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:33422_solr,127.0.0.1:38004_solr&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1283653 INFO  (OverseerThreadFactory-4994-thread-1-processing-n:127.0.0.1:33422_solr) [n:127.0.0.1:33422_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple1
   [junit4]   2> 1283770 DEBUG (OverseerThreadFactory-4994-thread-1-processing-n:127.0.0.1:33422_solr) [n:127.0.0.1:33422_solr    ] o.a.s.c.s.c.a.PolicyHelper Creating a new session
   [junit4]   2> 1283772 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1283774 INFO  (qtp1998724067-11233) [n:127.0.0.1:38004_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1283776 INFO  (qtp510530756-11246) [n:127.0.0.1:40251_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1283776 DEBUG (OverseerThreadFactory-4994-thread-1-processing-n:127.0.0.1:33422_solr) [n:127.0.0.1:33422_solr    ] o.a.s.c.s.c.a.PolicyHelper New session created
   [junit4]   2> 1283781 DEBUG (OverseerThreadFactory-4994-thread-1-processing-n:127.0.0.1:33422_solr) [n:127.0.0.1:33422_solr    ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 229896028 sessionWrapper.createTime 229896024186989, this.sessionWrapper.createTime 229896024186989
   [junit4]   2> 1283784 INFO  (OverseerStateUpdate-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [n:127.0.0.1:33422_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:38004/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1283787 INFO  (OverseerStateUpdate-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [n:127.0.0.1:33422_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testSimple1_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:33422/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1283793 INFO  (OverseerStateUpdate-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [n:127.0.0.1:33422_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:33422/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1283796 INFO  (OverseerStateUpdate-72072660294500359-127.0.0.1:33422_solr-n_0000000000) [n:127.0.0.1:33422_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testSimple1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"testSimple1_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:38004/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1283973 DEBUG (ScheduledTrigger-4992-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1283999 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr    x:testSimple1_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1283999 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr    x:testSimple1_shard1_replica_n2] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1284000 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr    x:testSimple1_shard2_replica_n5] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n5&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1284000 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr    x:testSimple1_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1284001 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr    x:testSimple1_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n6&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1284974 DEBUG (ScheduledTrigger-4992-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1285025 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1285027 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1285027 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1285031 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1285052 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 1285055 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1285055 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n1' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 1285055 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n5] Schema name=minimal
   [junit4]   2> 1285056 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard1.replica_n1' (registry 'solr.core.testSimple1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1285057 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 1285058 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1285058 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n5' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 1285059 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard2.replica_n5' (registry 'solr.core.testSimple1.shard2.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1285060 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1285060 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n2' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 1285061 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard1.replica_n2' (registry 'solr.core.testSimple1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1285069 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 1285077 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1285077 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n6' using configuration from collection testSimple1, trusted=true
   [junit4]   2> 1285078 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.testSimple1.shard2.replica_n6' (registry 'solr.core.testSimple1.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cbd7fc
   [junit4]   2> 1285082 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:32931/data
   [junit4]   2> 1285082 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:32931/data
   [junit4]   2> 1285082 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1285082 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1285082 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:32931/data
   [junit4]   2> 1285082 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1285082 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n5] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-001/node2/testSimple1_shard2_replica_n5], dataDir=[hdfs://localhost:32931/data/testSimple1/core_node7/data/]
   [junit4]   2> 1285082 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-001/node2/testSimple1_shard1_replica_n2], dataDir=[hdfs://localhost:32931/data/testSimple1/core_node4/data/]
   [junit4]   2> 1285084 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-001/node1/testSimple1_shard1_replica_n1], dataDir=[hdfs://localhost:32931/data/testSimple1/core_node3/data/]
   [junit4]   2> 1285084 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:32931/data
   [junit4]   2> 1285084 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1285084 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n6] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_2102D9FBAE552B33-001/tempDir-001/node1/testSimple1_shard2_replica_n6], dataDir=[hdfs://localhost:32931/data/testSimple1/core_node8/data/]
   [junit4]   2> 1285086 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node3/data/snapshot_metadata
   [junit4]   2> 1285091 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node4/data/snapshot_metadata
   [junit4]   2> 1285097 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node7/data/snapshot_metadata
   [junit4]   2> 1285102 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node8/data/snapshot_metadata
   [junit4]   2> 1285151 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node4/data
   [junit4]   2> 1285152 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node8/data
   [junit4]   2> 1285157 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node3/data
   [junit4]   2> 1285168 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node7/data
   [junit4]   2> 1285220 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node8/data/index
   [junit4]   2> 1285229 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node4/data/index
   [junit4]   2> 1285230 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node3/data/index
   [junit4]   2> 1285242 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:32931/data/testSimple1/core_node7/data/index
   [junit4]   2> 1285451 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46596 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-8fcc56f1-43aa-49c8-8f4d-590db7a34d2e:NORMAL:127.0.0.1:39496|RBW], ReplicaUC[[DISK]DS-ed3ac862-87c1-453d-b247-a94c618c1eb8:NORMAL:127.0.0.1:46596|RBW]]} size 0
   [junit4]   2> 1285451 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39496 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ed3ac862-87c1-453d-b247-a94c618c1eb8:NORMAL:127.0.0.1:46596|RBW], ReplicaUC[[DISK]DS-439b6082-eedc-4d5e-904a-79112a7c7115:NORMAL:127.0.0.1:39496|FINALIZED]]} size 0
   [junit4]   2> 1285453 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39496 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bb61c32-fbb2-4fe7-ba65-25815522d518:NORMAL:127.0.0.1:46596|RBW], ReplicaUC[[DISK]DS-439b6082-eedc-4d5e-904a-79112a7c7115:NORMAL:127.0.0.1:39496|RBW]]} size 0
   [junit4]   2> 1285454 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46596 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bb61c32-fbb2-4fe7-ba65-25815522d518:NORMAL:127.0.0.1:46596|RBW], ReplicaUC[[DISK]DS-8fcc56f1-43aa-49c8-8f4d-590db7a34d2e:NORMAL:127.0.0.1:39496|RBW]]} size 0
   [junit4]   2> 1285462 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46596 is added to blk_1073741825_1001 size 69
   [junit4]   2> 1285463 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39496 is added to blk_1073741826_1002 size 69
   [junit4]   2> 1285467 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46596 is added to blk_1073741828_1004 size 69
   [junit4]   2> 1285468 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39496 is added to blk_1073741827_1003 size 69
   [junit4]   2> 1285671 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1285671 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1285671 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 1285697 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1285697 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1285697 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 1285713 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1285714 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1285714 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 1285720 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1285720 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1285720 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 1285743 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1285743 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1285743 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1285743 INFO  (qtp1998724067-11218) [n:127.0.0.1:38004_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1285755 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1285755 INFO  (qtp1006018741-11219) [n:127.0.0.1:33422_solr c:testSimple1 s:shard1 r:core_node4 x:testSimple1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1285774 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1285774 INFO  (qtp1998724067-11229) [n:127.0.0.1:38004_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1285805 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.s.SolrIndexSearcher Opening [Searcher@66d56ecc[testSimple1_shard2_replica_n5] main]
   [junit4]   2> 1285816 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1285817 INFO  (qtp1006018741-11217) [n:127.0.0.1:33422_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n5] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperSt

[...truncated too long message...]

tateIsStale-seed#[857D0EF42B9A0198]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [stale_state_test_col] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:35193/solr/stale_state_test_col_shard1_replica_n1: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/stale_state_test_col_shard1_replica_n1/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/stale_state_test_col_shard1_replica_n1/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/stale_state_test_col_shard1_replica_n1/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 0
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node4 rstate:active live:true
   [junit4]   1> replica:core_node6 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> 15:05:16.512 [qtp1769471760-498] ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: Unknown action: foobar
   [junit4]   1> at org.apache.solr.handler.admin.ConfigSetsHandler.handleRequestBody(ConfigSetsHandler.java:109)
   [junit4]   1> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   1> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
   [junit4]   1> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   1> at org.eclipse.jetty.server.Server.handle(Server.java:531)
   [junit4]   1> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
   [junit4]   1> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   1> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   1> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   1> at java.lang.Thread.run(Thread.java:748)
   [junit4]   1>
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrClientTest -Dtests.method=testNonRetryableRequests -Dtests.seed=857D0EF42B9A0198 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=en-ZA -Dtests.timezone=America/Nipigon -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 4.23s J0 | CloudSolrClientTest.testNonRetryableRequests <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1> but was:<0>
   [junit4]    > at __randomizedtesting.SeedInfo.seed([857D0EF42B9A0198:2D93C967D5D37594]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests(CloudSolrClientTest.java:550)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   1> 15:05:19.295 [SUITE-CloudSolrClientTest-seed#[857D0EF42B9A0198]-worker] ERROR org.apache.zookeeper.server.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.impl.CloudSolrClientTest_857D0EF42B9A0198-001
   [junit4]   2> Oct 09, 2018 3:05:19 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=en-ZA, timezone=America/Nipigon
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=263739400,total=425197568
   [junit4]   2> NOTE: All tests run in this JVM: [TestJavaBinCodec, SolrExampleBinaryTest, MathExpressionTest, JdbcDriverTest, TestZkConfigManager, CloudSolrClientTest]
   [junit4] Completed [78/157 (1!)] on J0 in 60.87s, 16 tests, 2 failures <<< FAILURES!

[...truncated 45095 lines...]
-ecj-javadoc-lint-tests:
    [mkdir] Created dir: /tmp/ecj312598733
 [ecj-lint] Compiling 164 source files to /tmp/ecj312598733
 [ecj-lint] invalid Class-Path header in manifest of jar file: /x1/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: /x1/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] ----------
 [ecj-lint] 1. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleTests.java (at line 72)
 [ecj-lint] import org.apache.solr.common.util.Utils;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.common.util.Utils is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 2. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java (at line 2961)
 [ecj-lint] new SolrJSONWriter(writer)
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 3. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/common/util/TestSolrJsonWriter.java (at line 45)
 [ecj-lint] new SolrJSONWriter(writer)
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] 3 problems (1 error, 2 warnings)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:633: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:101: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build.xml:680: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2092: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2125: Compile failed; see the compiler error output for details.

Total time: 111 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 936 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/936/

4 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Expected new active leader null Live Nodes: [127.0.0.1:37407_solr, 127.0.0.1:40486_solr, 127.0.0.1:42446_solr] Last available state: DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/13)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"raceDeleteReplica_false_shard1_replica_n1",           "base_url":"https://127.0.0.1:44800/solr",           "node_name":"127.0.0.1:44800_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node6":{           "core":"raceDeleteReplica_false_shard1_replica_n5",           "base_url":"https://127.0.0.1:44800/solr",           "node_name":"127.0.0.1:44800_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected new active leader
null
Live Nodes: [127.0.0.1:37407_solr, 127.0.0.1:40486_solr, 127.0.0.1:42446_solr]
Last available state: DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/13)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"raceDeleteReplica_false_shard1_replica_n1",
          "base_url":"https://127.0.0.1:44800/solr",
          "node_name":"127.0.0.1:44800_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node6":{
          "core":"raceDeleteReplica_false_shard1_replica_n5",
          "base_url":"https://127.0.0.1:44800/solr",
          "node_name":"127.0.0.1:44800_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([B1CBCBCAE4CFFE6F:DBDDAA1A8C3DB4A5]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:333)
        at org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:229)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:36224_solr, 127.0.0.1:39963_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/18)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "core":"testSimple1_shard1_replica_n1",           "base_url":"https://127.0.0.1:39963/solr",           "node_name":"127.0.0.1:39963_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node12":{           "core":"testSimple1_shard1_replica_n11",           "base_url":"https://127.0.0.1:39329/solr",           "node_name":"127.0.0.1:39329_solr",           "state":"down",           "type":"NRT",           "force_set_state":"false"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node8":{           "core":"testSimple1_shard2_replica_n6",           "base_url":"https://127.0.0.1:39963/solr",           "node_name":"127.0.0.1:39963_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false",           "leader":"true"},         "core_node10":{           "core":"testSimple1_shard2_replica_n9",           "base_url":"https://127.0.0.1:39963/solr",           "node_name":"127.0.0.1:39963_solr",           "state":"active",           "type":"NRT",           "force_set_state":"false"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:36224_solr, 127.0.0.1:39963_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/18)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "core":"testSimple1_shard1_replica_n1",
          "base_url":"https://127.0.0.1:39963/solr",
          "node_name":"127.0.0.1:39963_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node12":{
          "core":"testSimple1_shard1_replica_n11",
          "base_url":"https://127.0.0.1:39329/solr",
          "node_name":"127.0.0.1:39329_solr",
          "state":"down",
          "type":"NRT",
          "force_set_state":"false"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node8":{
          "core":"testSimple1_shard2_replica_n6",
          "base_url":"https://127.0.0.1:39963/solr",
          "node_name":"127.0.0.1:39963_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false",
          "leader":"true"},
        "core_node10":{
          "core":"testSimple1_shard2_replica_n9",
          "base_url":"https://127.0.0.1:39963/solr",
          "node_name":"127.0.0.1:39963_solr",
          "state":"active",
          "type":"NRT",
          "force_set_state":"false"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([B1CBCBCAE4CFFE6F:8978EF34C33C2ABE]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple1 null Live Nodes: [127.0.0.1:33377_solr, 127.0.0.1:45300_solr] Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/19)={   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{         "core_node3":{           "dataDir":"hdfs://localhost:37129/data/testSimple1/core_node3/data/",           "base_url":"http://127.0.0.1:33377/solr",           "node_name":"127.0.0.1:33377_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost:37129/data/testSimple1/core_node3/data/tlog",           "core":"testSimple1_shard1_replica_n1",           "shared_storage":"true",           "state":"active",           "leader":"true"},         "core_node5":{           "dataDir":"hdfs://localhost:37129/data/testSimple1/core_node5/data/",           "base_url":"http://127.0.0.1:33377/solr",           "node_name":"127.0.0.1:33377_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost:37129/data/testSimple1/core_node5/data/tlog",           "core":"testSimple1_shard1_replica_n2",           "shared_storage":"true",           "state":"active"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{         "core_node7":{           "dataDir":"hdfs://localhost:37129/data/testSimple1/core_node7/data/",           "base_url":"http://127.0.0.1:33602/solr",           "node_name":"127.0.0.1:33602_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost:37129/data/testSimple1/core_node7/data/tlog",           "core":"testSimple1_shard2_replica_n4",           "shared_storage":"true",           "state":"down"},         "core_node8":{           "dataDir":"hdfs://localhost:37129/data/testSimple1/core_node8/data/",           "base_url":"http://127.0.0.1:33377/solr",           "node_name":"127.0.0.1:33377_solr",           "type":"NRT",           "force_set_state":"false",           "ulogDir":"hdfs://localhost:37129/data/testSimple1/core_node8/data/tlog",           "core":"testSimple1_shard2_replica_n6",           "shared_storage":"true",           "state":"active",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple1
null
Live Nodes: [127.0.0.1:33377_solr, 127.0.0.1:45300_solr]
Last available state: DocCollection(testSimple1//collections/testSimple1/state.json/19)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node3":{
          "dataDir":"hdfs://localhost:37129/data/testSimple1/core_node3/data/",
          "base_url":"http://127.0.0.1:33377/solr",
          "node_name":"127.0.0.1:33377_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost:37129/data/testSimple1/core_node3/data/tlog",
          "core":"testSimple1_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node5":{
          "dataDir":"hdfs://localhost:37129/data/testSimple1/core_node5/data/",
          "base_url":"http://127.0.0.1:33377/solr",
          "node_name":"127.0.0.1:33377_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost:37129/data/testSimple1/core_node5/data/tlog",
          "core":"testSimple1_shard1_replica_n2",
          "shared_storage":"true",
          "state":"active"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node7":{
          "dataDir":"hdfs://localhost:37129/data/testSimple1/core_node7/data/",
          "base_url":"http://127.0.0.1:33602/solr",
          "node_name":"127.0.0.1:33602_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost:37129/data/testSimple1/core_node7/data/tlog",
          "core":"testSimple1_shard2_replica_n4",
          "shared_storage":"true",
          "state":"down"},
        "core_node8":{
          "dataDir":"hdfs://localhost:37129/data/testSimple1/core_node8/data/",
          "base_url":"http://127.0.0.1:33377/solr",
          "node_name":"127.0.0.1:33377_solr",
          "type":"NRT",
          "force_set_state":"false",
          "ulogDir":"hdfs://localhost:37129/data/testSimple1/core_node8/data/tlog",
          "core":"testSimple1_shard2_replica_n6",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at __randomizedtesting.SeedInfo.seed([B1CBCBCAE4CFFE6F:8978EF34C33C2ABE]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
        at org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([15C9B5F3F64008AD:BD27726008097CA1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testNonRetryableRequests(CloudSolrClientTest.java:550)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13957 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> 1512999 INFO  (SUITE-DeleteReplicaTest-seed#[B1CBCBCAE4CFFE6F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/init-core-data-001
   [junit4]   2> 1513000 WARN  (SUITE-DeleteReplicaTest-seed#[B1CBCBCAE4CFFE6F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 1513000 INFO  (SUITE-DeleteReplicaTest-seed#[B1CBCBCAE4CFFE6F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1513002 INFO  (SUITE-DeleteReplicaTest-seed#[B1CBCBCAE4CFFE6F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1513003 INFO  (SUITE-DeleteReplicaTest-seed#[B1CBCBCAE4CFFE6F]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001
   [junit4]   2> 1513003 INFO  (SUITE-DeleteReplicaTest-seed#[B1CBCBCAE4CFFE6F]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1513004 INFO  (Thread-4261) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1513004 INFO  (Thread-4261) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1513006 ERROR (Thread-4261) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1513104 INFO  (SUITE-DeleteReplicaTest-seed#[B1CBCBCAE4CFFE6F]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:41902
   [junit4]   2> 1513106 INFO  (zkConnectionManagerCallback-3606-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513114 INFO  (jetty-launcher-3603-thread-4) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1513114 INFO  (jetty-launcher-3603-thread-1) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1513116 INFO  (jetty-launcher-3603-thread-2) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1513120 INFO  (jetty-launcher-3603-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1513120 INFO  (jetty-launcher-3603-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1513120 INFO  (jetty-launcher-3603-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1513120 INFO  (jetty-launcher-3603-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@538ac62a{/solr,null,AVAILABLE}
   [junit4]   2> 1513123 INFO  (jetty-launcher-3603-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1513123 INFO  (jetty-launcher-3603-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1513123 INFO  (jetty-launcher-3603-thread-4) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1513123 INFO  (jetty-launcher-3603-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@220397a0{/solr,null,AVAILABLE}
   [junit4]   2> 1513124 INFO  (jetty-launcher-3603-thread-3) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1513124 INFO  (jetty-launcher-3603-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1513124 INFO  (jetty-launcher-3603-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1513125 INFO  (jetty-launcher-3603-thread-3) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1513125 INFO  (jetty-launcher-3603-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@478c3e4a{/solr,null,AVAILABLE}
   [junit4]   2> 1513125 INFO  (jetty-launcher-3603-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3be354a4{SSL,[ssl, http/1.1]}{127.0.0.1:37407}
   [junit4]   2> 1513125 INFO  (jetty-launcher-3603-thread-3) [    ] o.e.j.s.Server Started @1513184ms
   [junit4]   2> 1513125 INFO  (jetty-launcher-3603-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37407}
   [junit4]   2> 1513126 ERROR (jetty-launcher-3603-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1513126 INFO  (jetty-launcher-3603-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1513126 INFO  (jetty-launcher-3603-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1513126 INFO  (jetty-launcher-3603-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1513126 INFO  (jetty-launcher-3603-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1513126 INFO  (jetty-launcher-3603-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T23:48:56.679Z
   [junit4]   2> 1513138 INFO  (jetty-launcher-3603-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@ffe1a10{SSL,[ssl, http/1.1]}{127.0.0.1:42446}
   [junit4]   2> 1513138 INFO  (jetty-launcher-3603-thread-4) [    ] o.e.j.s.Server Started @1513197ms
   [junit4]   2> 1513138 INFO  (jetty-launcher-3603-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42446}
   [junit4]   2> 1513138 ERROR (jetty-launcher-3603-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1513138 INFO  (jetty-launcher-3603-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1513139 INFO  (jetty-launcher-3603-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1513139 INFO  (jetty-launcher-3603-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1513139 INFO  (jetty-launcher-3603-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1513139 INFO  (jetty-launcher-3603-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T23:48:56.692Z
   [junit4]   2> 1513140 INFO  (jetty-launcher-3603-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1df0f269{SSL,[ssl, http/1.1]}{127.0.0.1:44800}
   [junit4]   2> 1513140 INFO  (jetty-launcher-3603-thread-1) [    ] o.e.j.s.Server Started @1513199ms
   [junit4]   2> 1513140 INFO  (jetty-launcher-3603-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44800}
   [junit4]   2> 1513140 ERROR (jetty-launcher-3603-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1513140 INFO  (jetty-launcher-3603-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1513140 INFO  (jetty-launcher-3603-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1513140 INFO  (jetty-launcher-3603-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1513140 INFO  (jetty-launcher-3603-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1513140 INFO  (jetty-launcher-3603-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T23:48:56.693Z
   [junit4]   2> 1513141 INFO  (zkConnectionManagerCallback-3608-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513141 INFO  (jetty-launcher-3603-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1513145 INFO  (jetty-launcher-3603-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1513145 INFO  (jetty-launcher-3603-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1513145 INFO  (jetty-launcher-3603-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1513146 INFO  (jetty-launcher-3603-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3e495b58{/solr,null,AVAILABLE}
   [junit4]   2> 1513146 INFO  (jetty-launcher-3603-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@77984a1d{SSL,[ssl, http/1.1]}{127.0.0.1:40486}
   [junit4]   2> 1513147 INFO  (jetty-launcher-3603-thread-2) [    ] o.e.j.s.Server Started @1513205ms
   [junit4]   2> 1513147 INFO  (jetty-launcher-3603-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40486}
   [junit4]   2> 1513147 ERROR (jetty-launcher-3603-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1513147 INFO  (jetty-launcher-3603-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1513147 INFO  (jetty-launcher-3603-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1513147 INFO  (jetty-launcher-3603-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1513147 INFO  (jetty-launcher-3603-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1513147 INFO  (jetty-launcher-3603-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-09T23:48:56.700Z
   [junit4]   2> 1513148 INFO  (zkConnectionManagerCallback-3610-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513149 INFO  (zkConnectionManagerCallback-3614-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513149 INFO  (jetty-launcher-3603-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1513150 INFO  (jetty-launcher-3603-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1513154 INFO  (zkConnectionManagerCallback-3612-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513175 INFO  (jetty-launcher-3603-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1513325 INFO  (jetty-launcher-3603-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41902/solr
   [junit4]   2> 1513327 INFO  (zkConnectionManagerCallback-3618-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513329 INFO  (zkConnectionManagerCallback-3620-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513345 INFO  (jetty-launcher-3603-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41902/solr
   [junit4]   2> 1513347 INFO  (zkConnectionManagerCallback-3626-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513349 INFO  (jetty-launcher-3603-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41902/solr
   [junit4]   2> 1513350 INFO  (zkConnectionManagerCallback-3630-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513350 INFO  (zkConnectionManagerCallback-3632-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513352 INFO  (zkConnectionManagerCallback-3634-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513371 INFO  (jetty-launcher-3603-thread-4) [n:127.0.0.1:42446_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42446_solr
   [junit4]   2> 1513373 INFO  (jetty-launcher-3603-thread-4) [n:127.0.0.1:42446_solr    ] o.a.s.c.Overseer Overseer (id=72074877540565002-127.0.0.1:42446_solr-n_0000000000) starting
   [junit4]   2> 1513387 INFO  (zkConnectionManagerCallback-3643-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513390 INFO  (jetty-launcher-3603-thread-4) [n:127.0.0.1:42446_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41902/solr ready
   [junit4]   2> 1513392 INFO  (OverseerStateUpdate-72074877540565002-127.0.0.1:42446_solr-n_0000000000) [n:127.0.0.1:42446_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:42446_solr
   [junit4]   2> 1513400 INFO  (jetty-launcher-3603-thread-4) [n:127.0.0.1:42446_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42446_solr
   [junit4]   2> 1513412 INFO  (zkCallback-3633-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1513415 INFO  (zkCallback-3642-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1513437 INFO  (jetty-launcher-3603-thread-4) [n:127.0.0.1:42446_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1513484 INFO  (jetty-launcher-3603-thread-2) [n:127.0.0.1:40486_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1513487 INFO  (jetty-launcher-3603-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41902/solr
   [junit4]   2> 1513487 INFO  (jetty-launcher-3603-thread-4) [n:127.0.0.1:42446_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42446.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513495 INFO  (jetty-launcher-3603-thread-3) [n:127.0.0.1:37407_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1513495 INFO  (jetty-launcher-3603-thread-2) [n:127.0.0.1:40486_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1513496 INFO  (jetty-launcher-3603-thread-2) [n:127.0.0.1:40486_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40486_solr
   [junit4]   2> 1513496 INFO  (zkConnectionManagerCallback-3648-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513497 INFO  (zkCallback-3642-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1513500 INFO  (zkCallback-3619-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1513503 INFO  (zkCallback-3633-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1513504 INFO  (jetty-launcher-3603-thread-4) [n:127.0.0.1:42446_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42446.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513504 INFO  (jetty-launcher-3603-thread-4) [n:127.0.0.1:42446_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42446.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513504 INFO  (zkCallback-3631-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1513506 INFO  (jetty-launcher-3603-thread-4) [n:127.0.0.1:42446_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node2/.
   [junit4]   2> 1513508 INFO  (jetty-launcher-3603-thread-3) [n:127.0.0.1:37407_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1513508 INFO  (jetty-launcher-3603-thread-3) [n:127.0.0.1:37407_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37407_solr
   [junit4]   2> 1513509 INFO  (zkCallback-3633-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1513509 INFO  (zkCallback-3642-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1513516 INFO  (zkCallback-3631-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1513516 INFO  (zkCallback-3619-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1513522 INFO  (zkConnectionManagerCallback-3650-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513530 INFO  (jetty-launcher-3603-thread-1) [n:127.0.0.1:44800_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1513535 INFO  (jetty-launcher-3603-thread-1) [n:127.0.0.1:44800_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1513535 INFO  (jetty-launcher-3603-thread-1) [n:127.0.0.1:44800_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44800_solr
   [junit4]   2> 1513537 INFO  (zkCallback-3642-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1513537 INFO  (zkCallback-3631-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1513537 INFO  (zkCallback-3619-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1513537 INFO  (zkCallback-3633-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1513544 INFO  (zkConnectionManagerCallback-3655-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513544 INFO  (zkConnectionManagerCallback-3662-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513545 INFO  (zkCallback-3649-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1513546 INFO  (jetty-launcher-3603-thread-2) [n:127.0.0.1:40486_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1513546 INFO  (jetty-launcher-3603-thread-3) [n:127.0.0.1:37407_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1513548 INFO  (jetty-launcher-3603-thread-2) [n:127.0.0.1:40486_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41902/solr ready
   [junit4]   2> 1513548 INFO  (jetty-launcher-3603-thread-3) [n:127.0.0.1:37407_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41902/solr ready
   [junit4]   2> 1513548 INFO  (jetty-launcher-3603-thread-2) [n:127.0.0.1:40486_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1513555 INFO  (jetty-launcher-3603-thread-3) [n:127.0.0.1:37407_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1513567 INFO  (zkConnectionManagerCallback-3669-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513568 INFO  (jetty-launcher-3603-thread-1) [n:127.0.0.1:44800_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1513569 INFO  (jetty-launcher-3603-thread-1) [n:127.0.0.1:44800_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41902/solr ready
   [junit4]   2> 1513570 INFO  (jetty-launcher-3603-thread-1) [n:127.0.0.1:44800_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1513573 INFO  (jetty-launcher-3603-thread-2) [n:127.0.0.1:40486_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40486.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513580 INFO  (jetty-launcher-3603-thread-3) [n:127.0.0.1:37407_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37407.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513610 INFO  (jetty-launcher-3603-thread-2) [n:127.0.0.1:40486_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40486.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513610 INFO  (jetty-launcher-3603-thread-2) [n:127.0.0.1:40486_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40486.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513612 INFO  (jetty-launcher-3603-thread-2) [n:127.0.0.1:40486_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node4/.
   [junit4]   2> 1513616 INFO  (jetty-launcher-3603-thread-1) [n:127.0.0.1:44800_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44800.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513617 INFO  (jetty-launcher-3603-thread-3) [n:127.0.0.1:37407_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37407.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513618 INFO  (jetty-launcher-3603-thread-3) [n:127.0.0.1:37407_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37407.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513620 INFO  (jetty-launcher-3603-thread-3) [n:127.0.0.1:37407_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node3/.
   [junit4]   2> 1513634 INFO  (jetty-launcher-3603-thread-1) [n:127.0.0.1:44800_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44800.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513635 INFO  (jetty-launcher-3603-thread-1) [n:127.0.0.1:44800_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44800.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1513637 INFO  (jetty-launcher-3603-thread-1) [n:127.0.0.1:44800_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node1/.
   [junit4]   2> 1513780 INFO  (zkConnectionManagerCallback-3672-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513785 INFO  (zkConnectionManagerCallback-3677-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1513786 INFO  (SUITE-DeleteReplicaTest-seed#[B1CBCBCAE4CFFE6F]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1513787 INFO  (SUITE-DeleteReplicaTest-seed#[B1CBCBCAE4CFFE6F]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41902/solr ready
   [junit4]   2> 1513848 INFO  (TEST-DeleteReplicaTest.deleteReplicaByCountForAllShards-seed#[B1CBCBCAE4CFFE6F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting deleteReplicaByCountForAllShards
   [junit4]   2> 1513865 INFO  (qtp2077882247-15548) [n:127.0.0.1:42446_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&name=deleteByCountNew&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1513868 INFO  (OverseerThreadFactory-6533-thread-1-processing-n:127.0.0.1:42446_solr) [n:127.0.0.1:42446_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection deleteByCountNew
   [junit4]   2> 1513987 INFO  (qtp2077882247-15549) [n:127.0.0.1:42446_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1514003 INFO  (qtp1452258313-15536) [n:127.0.0.1:44800_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1514018 INFO  (qtp383919987-15555) [n:127.0.0.1:37407_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1514033 INFO  (qtp857090470-15567) [n:127.0.0.1:40486_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1514040 INFO  (OverseerStateUpdate-72074877540565002-127.0.0.1:42446_solr-n_0000000000) [n:127.0.0.1:42446_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteByCountNew",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deleteByCountNew_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:40486/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1514043 INFO  (OverseerStateUpdate-72074877540565002-127.0.0.1:42446_solr-n_0000000000) [n:127.0.0.1:42446_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteByCountNew",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"deleteByCountNew_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37407/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1514046 INFO  (OverseerStateUpdate-72074877540565002-127.0.0.1:42446_solr-n_0000000000) [n:127.0.0.1:42446_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteByCountNew",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"deleteByCountNew_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:44800/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1514050 INFO  (OverseerStateUpdate-72074877540565002-127.0.0.1:42446_solr-n_0000000000) [n:127.0.0.1:42446_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"deleteByCountNew",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"deleteByCountNew_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:42446/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1514263 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr    x:deleteByCountNew_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=deleteByCountNew_shard2_replica_n6&action=CREATE&numShards=2&collection=deleteByCountNew&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1514263 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr    x:deleteByCountNew_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deleteByCountNew_shard1_replica_n1&action=CREATE&numShards=2&collection=deleteByCountNew&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1514264 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr    x:deleteByCountNew_shard2_replica_n6] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1514264 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr    x:deleteByCountNew_shard2_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=deleteByCountNew_shard2_replica_n4&action=CREATE&numShards=2&collection=deleteByCountNew&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1514269 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr    x:deleteByCountNew_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=deleteByCountNew_shard1_replica_n2&action=CREATE&numShards=2&collection=deleteByCountNew&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1515286 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1515288 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1515289 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1515303 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1515307 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.s.IndexSchema [deleteByCountNew_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 1515308 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.s.IndexSchema [deleteByCountNew_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 1515311 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1515311 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'deleteByCountNew_shard1_replica_n1' using configuration from collection deleteByCountNew, trusted=true
   [junit4]   2> 1515312 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40486.solr.core.deleteByCountNew.shard1.replica_n1' (registry 'solr.core.deleteByCountNew.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1515312 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.c.SolrCore [[deleteByCountNew_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node4/deleteByCountNew_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node4/./deleteByCountNew_shard1_replica_n1/data/]
   [junit4]   2> 1515313 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.s.IndexSchema [deleteByCountNew_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 1515313 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1515313 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'deleteByCountNew_shard2_replica_n4' using configuration from collection deleteByCountNew, trusted=true
   [junit4]   2> 1515314 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44800.solr.core.deleteByCountNew.shard2.replica_n4' (registry 'solr.core.deleteByCountNew.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1515314 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.c.SolrCore [[deleteByCountNew_shard2_replica_n4] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node1/deleteByCountNew_shard2_replica_n4], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node1/./deleteByCountNew_shard2_replica_n4/data/]
   [junit4]   2> 1515317 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.s.IndexSchema [deleteByCountNew_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 1515320 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1515320 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'deleteByCountNew_shard1_replica_n2' using configuration from collection deleteByCountNew, trusted=true
   [junit4]   2> 1515320 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37407.solr.core.deleteByCountNew.shard1.replica_n2' (registry 'solr.core.deleteByCountNew.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1515321 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.SolrCore [[deleteByCountNew_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node3/deleteByCountNew_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node3/./deleteByCountNew_shard1_replica_n2/data/]
   [junit4]   2> 1515325 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1515325 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'deleteByCountNew_shard2_replica_n6' using configuration from collection deleteByCountNew, trusted=true
   [junit4]   2> 1515325 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42446.solr.core.deleteByCountNew.shard2.replica_n6' (registry 'solr.core.deleteByCountNew.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7eaf5b61
   [junit4]   2> 1515325 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.SolrCore [[deleteByCountNew_shard2_replica_n6] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node2/deleteByCountNew_shard2_replica_n6], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest_B1CBCBCAE4CFFE6F-001/tempDir-001/node2/./deleteByCountNew_shard2_replica_n6/data/]
   [junit4]   2> 1515418 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1515418 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1515418 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1515419 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1515434 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1515434 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1515435 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1515435 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1515435 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1515435 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1515436 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1515436 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1515437 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@362c4811[deleteByCountNew_shard1_replica_n2] main]
   [junit4]   2> 1515438 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@308d75c6[deleteByCountNew_shard2_replica_n6] main]
   [junit4]   2> 1515439 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1515439 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1515439 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1515439 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5d2e6e7d[deleteByCountNew_shard1_replica_n1] main]
   [junit4]   2> 1515440 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1515440 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1515440 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1515440 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1515441 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1515441 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613893666334572544
   [junit4]   2> 1515441 INFO  (searcherExecutor-6555-thread-1-processing-n:127.0.0.1:37407_solr x:deleteByCountNew_shard1_replica_n2 c:deleteByCountNew s:shard1 r:core_node5) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.SolrCore [deleteByCountNew_shard1_replica_n2] Registered new searcher Searcher@362c4811[deleteByCountNew_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1515441 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613893666334572544
   [junit4]   2> 1515442 INFO  (searcherExecutor-6556-thread-1-processing-n:127.0.0.1:42446_solr x:deleteByCountNew_shard2_replica_n6 c:deleteByCountNew s:shard2 r:core_node8) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.SolrCore [deleteByCountNew_shard2_replica_n6] Registered new searcher Searcher@308d75c6[deleteByCountNew_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1515442 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1515442 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1515444 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1515444 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1515445 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@7df1858e[deleteByCountNew_shard2_replica_n4] main]
   [junit4]   2> 1515446 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1515446 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613893666339815424
   [junit4]   2> 1515447 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1515447 INFO  (searcherExecutor-6553-thread-1-processing-n:127.0.0.1:40486_solr x:deleteByCountNew_shard1_replica_n1 c:deleteByCountNew s:shard1 r:core_node3) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.c.SolrCore [deleteByCountNew_shard1_replica_n1] Registered new searcher Searcher@5d2e6e7d[deleteByCountNew_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1515447 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1515448 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1515448 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613893666341912576
   [junit4]   2> 1515449 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteByCountNew/terms/shard2 to Terms{values={core_node8=0}, version=0}
   [junit4]   2> 1515451 INFO  (searcherExecutor-6554-thread-1-processing-n:127.0.0.1:44800_solr x:deleteByCountNew_shard2_replica_n4 c:deleteByCountNew s:shard2 r:core_node7) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.c.SolrCore [deleteByCountNew_shard2_replica_n4] Registered new searcher Searcher@7df1858e[deleteByCountNew_shard2_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1515456 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteByCountNew/terms/shard1 to Terms{values={core_node5=0}, version=0}
   [junit4]   2> 1515460 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteByCountNew/terms/shard2 to Terms{values={core_node7=0, core_node8=0}, version=1}
   [junit4]   2> 1515462 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/deleteByCountNew/terms/shard1 to Terms{values={core_node3=0, core_node5=0}, version=1}
   [junit4]   2> 1515467 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1515467 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1515467 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:42446/solr/deleteByCountNew_shard2_replica_n6/
   [junit4]   2> 1515468 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1515468 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1515468 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:37407/solr/deleteByCountNew_shard1_replica_n2/
   [junit4]   2> 1515469 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.u.PeerSync PeerSync: core=deleteByCountNew_shard2_replica_n6 url=https://127.0.0.1:42446/solr START replicas=[https://127.0.0.1:44800/solr/deleteByCountNew_shard2_replica_n4/] nUpdates=100
   [junit4]   2> 1515469 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.u.PeerSync PeerSync: core=deleteByCountNew_shard2_replica_n6 url=https://127.0.0.1:42446/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1515469 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=deleteByCountNew_shard1_replica_n2 url=https://127.0.0.1:37407/solr START replicas=[https://127.0.0.1:40486/solr/deleteByCountNew_shard1_replica_n1/] nUpdates=100
   [junit4]   2> 1515470 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=deleteByCountNew_shard1_replica_n2 url=https://127.0.0.1:37407/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1515475 INFO  (qtp1452258313-15538) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.c.S.Request [deleteByCountNew_shard2_replica_n4]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 1515475 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1515475 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1515476 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1515478 INFO  (qtp857090470-15570) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.c.S.Request [deleteByCountNew_shard1_replica_n1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1515478 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1515478 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1515478 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1515481 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:42446/solr/deleteByCountNew_shard2_replica_n6/ shard2
   [junit4]   2> 1515483 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:37407/solr/deleteByCountNew_shard1_replica_n2/ shard1
   [junit4]   2> 1515586 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1515589 INFO  (qtp383919987-15551) [n:127.0.0.1:37407_solr c:deleteByCountNew s:shard1 r:core_node5 x:deleteByCountNew_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=deleteByCountNew_shard1_replica_n2&action=CREATE&numShards=2&collection=deleteByCountNew&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1320
   [junit4]   2> 1515635 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1515637 INFO  (qtp2077882247-15546) [n:127.0.0.1:42446_solr c:deleteByCountNew s:shard2 r:core_node8 x:deleteByCountNew_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=deleteByCountNew_shard2_replica_n6&action=CREATE&numShards=2&collection=deleteByCountNew&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1374
   [junit4]   2> 1515737 INFO  (zkCallback-3633-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCountNew/state.json] for collection [deleteByCountNew] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1515737 INFO  (zkCallback-3631-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCountNew/state.json] for collection [deleteByCountNew] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1516475 INFO  (qtp1452258313-15532) [n:127.0.0.1:44800_solr c:deleteByCountNew s:shard2 r:core_node7 x:deleteByCountNew_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=deleteByCountNew_shard2_replica_n4&action=CREATE&numShards=2&collection=deleteByCountNew&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2211
   [junit4]   2> 1516477 INFO  (qtp857090470-15541) [n:127.0.0.1:40486_solr c:deleteByCountNew s:shard1 r:core_node3 x:deleteByCountNew_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=deleteByCountNew_shard1_replica_n1&action=CREATE&numShards=2&collection=deleteByCountNew&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2214
   [junit4]   2> 1516480 INFO  (qtp2077882247-15548) [n:127.0.0.1:42446_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1516577 INFO  (zkCallback-3631-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCountNew/state.json] for collection [deleteByCountNew] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1516577 INFO  (zkCallback-3649-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCountNew/state.json] for collection [deleteByCountNew] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1516577 INFO  (zkCallback-3619-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCountNew/state.json] for collection [deleteByCountNew] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1516577 INFO  (zkCallback-3633-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/deleteByCountNew/state.json] for collection [deleteByCountNew] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1517481 INFO  (qtp2077882247-15548) [n:127.0.0.1:42446_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=deleteByCountNew&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=3615
   [junit4]   2> 1517483 INFO  (qtp2077882247-15547) [n:127.0.0.1:42446_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :deletereplica with params count=1&action=DELETEREPLICA&collection=deleteByCountNew&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1517485 INFO  (OverseerCollectionConfigSetProcessor-72074877540565002-127.0.0.1:42446_solr-n_0000000000) [n:127.0.0.1:42446_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1517487 INFO  (qtp857090470-15567) [n:127.0.0.1:40486_solr    x:deleteByCountNew_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.deleteByCountNew.shard1.replica_n1, tag=null
   [junit4]   2> 1517487 INFO  (qtp857090470-15567) [n:127.0.0.1:40486_solr    x:deleteByCountNew_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@40cbf3a3: rootName = solr_40486, domain = solr.core.deleteByCountNew.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.deleteByCountNew.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@60e1c830
   [junit4]   2> 1517508 INFO  (qtp857090470-15567) [n:127.0.0.1:40486_solr    x:deleteByCountNew_shard1_replica_n1] o.a.s.c.SolrCore [deleteByCountNew_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCor

[...truncated too long message...]

-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/null717000314
     [copy] Copying 239 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/null717000314
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/null717000314

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 52ms :: artifacts dl 3ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 119 minutes 0 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 937 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/937/

1 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting

Error Message:
Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason: <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>  </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at __randomizedtesting.SeedInfo.seed([49835F707737C5C:C62F099F04338C24]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting(CloudSolrClientTest.java:238)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>

</body>
</html>

        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more




Build Log:
[...truncated 16324 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrClientTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.impl.CloudSolrClientTest_49835F707737C5C-001/init-core-data-001
   [junit4]   1> 12:39:18.376 [Thread-142] ERROR org.apache.zookeeper.server.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   1> 12:39:18.577 [jetty-launcher-244-thread-1] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   1> 12:39:18.578 [jetty-launcher-244-thread-2] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   1> 12:39:18.582 [jetty-launcher-244-thread-3] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   1> 12:39:23.823 [TEST-CloudSolrClientTest.testShutdown-seed#[49835F707737C5C]-SendThread(ff01::114:33332)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   1> 12:39:24.960 [TEST-CloudSolrClientTest.testShutdown-seed#[49835F707737C5C]-SendThread(ff01::114:33332)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   1> 12:39:55.017 [TEST-CloudSolrClientTest.testRouting-seed#[49835F707737C5C]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [collection1] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 0
   [junit4]   1> 12:39:55.030 [TEST-CloudSolrClientTest.testRouting-seed#[49835F707737C5C]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [collection1] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 1
   [junit4]   1> 12:39:55.035 [TEST-CloudSolrClientTest.testRouting-seed#[49835F707737C5C]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [collection1] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 2
   [junit4]   1> 12:39:55.052 [TEST-CloudSolrClientTest.testRouting-seed#[49835F707737C5C]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [collection1] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 3
   [junit4]   1> 12:39:55.064 [TEST-CloudSolrClientTest.testRouting-seed#[49835F707737C5C]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [collection1] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 4
   [junit4]   1> 12:39:55.072 [TEST-CloudSolrClientTest.testRouting-seed#[49835F707737C5C]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [collection1] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 5
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrClientTest -Dtests.method=testRouting -Dtests.seed=49835F707737C5C -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=fr-CA -Dtests.timezone=America/Puerto_Rico -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   4.11s J1 | CloudSolrClientTest.testRouting <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]    > <head>
   [junit4]    > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]    > <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]    > </head>
   [junit4]    > <body><h2>HTTP ERROR 404</h2>
   [junit4]    > <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]    > <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > at __randomizedtesting.SeedInfo.seed([49835F707737C5C:C62F099F04338C24]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting(CloudSolrClientTest.java:238)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36861/solr/collection1_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]    > <head>
   [junit4]    > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]    > <title>Error 404 Can not find: /solr/collection1_shard2_replica_n2/update</title>
   [junit4]    > </head>
   [junit4]    > <body><h2>HTTP ERROR 404</h2>
   [junit4]    > <p>Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
   [junit4]    > <pre>    Can not find: /solr/collection1_shard2_replica_n2/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
   [junit4]    > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]    > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    > ... 1 more
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node4 rstate:down live:true
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node4 rstate:recovering live:true
   [junit4]   1> -
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node4 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> -
   [junit4]   1> replica:core_node4 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> 12:40:11.173 [TEST-CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale-seed#[49835F707737C5C]] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient - Request to collection [stale_state_test_col] failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:36861/solr/stale_state_test_col_shard1_replica_n1: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   1> <head>
   [junit4]   1> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
   [junit4]   1> <title>Error 404 Can not find: /solr/stale_state_test_col_shard1_replica_n1/update</title>
   [junit4]   1> </head>
   [junit4]   1> <body><h2>HTTP ERROR 404</h2>
   [junit4]   1> <p>Problem accessing /solr/stale_state_test_col_shard1_replica_n1/update. Reason:
   [junit4]   1> <pre>    Can not find: /solr/stale_state_test_col_shard1_replica_n1/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.11.v20180605</a><hr/>
   [junit4]   1>
   [junit4]   1> </body>
   [junit4]   1> </html>
   [junit4]   1> , retry? 0
   [junit4]   1> 12:40:14.399 [qtp1609339444-943] ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: Collection: boguscollectionname not found
   [junit4]   1> at org.apache.solr.handler.admin.ClusterStatus.getClusterStatus(ClusterStatus.java:103)
   [junit4]   1> at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$22(CollectionsHandler.java:858)
   [junit4]   1> at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1170)
   [junit4]   1> at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:257)
   [junit4]   1> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:245)
   [junit4]   1> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   1> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
   [junit4]   1> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   1> at org.eclipse.jetty.server.Server.handle(Server.java:531)
   [junit4]   1> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
   [junit4]   1> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   1> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   1> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   1> at java.lang.Thread.run(Thread.java:748)
   [junit4]   1>
   [junit4]   1> 12:40:14.403 [qtp1609339444-943] ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: Collection: boguscollectionname not found
   [junit4]   1> at org.apache.solr.handler.admin.ClusterStatus.getClusterStatus(ClusterStatus.java:103)
   [junit4]   1> at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$22(CollectionsHandler.java:858)
   [junit4]   1> at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1170)
   [junit4]   1> at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:257)
   [junit4]   1> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:245)
   [junit4]   1> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   1> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
   [junit4]   1> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   1> at org.eclipse.jetty.server.Server.handle(Server.java:531)
   [junit4]   1> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
   [junit4]   1> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   1> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   1> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   1> at java.lang.Thread.run(Thread.java:748)
   [junit4]   1>
   [junit4]   1> -
   [junit4]   1> replica:core_node3 rstate:active live:true
   [junit4]   1> replica:core_node5 rstate:active live:true
   [junit4]   1> replica:core_node6 rstate:active live:true
   [junit4]   1> no one is recoverying
   [junit4]   1> 12:40:19.786 [qtp714361647-949] ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: Unknown action: foobar
   [junit4]   1> at org.apache.solr.handler.admin.ConfigSetsHandler.handleRequestBody(ConfigSetsHandler.java:109)
   [junit4]   1> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   1> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
   [junit4]   1> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   1> at org.eclipse.jetty.server.Server.handle(Server.java:531)
   [junit4]   1> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
   [junit4]   1> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   1> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   1> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   1> at java.lang.Thread.run(Thread.java:748)
   [junit4]   1>
   [junit4]   1> 12:40:19.822 [qtp714361647-949] ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: Unknown action: foobar
   [junit4]   1> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:240)
   [junit4]   1> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   1> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
   [junit4]   1> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   1> at org.eclipse.jetty.server.Server.handle(Server.java:531)
   [junit4]   1> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
   [junit4]   1> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   1> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   1> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   1> at java.lang.Thread.run(Thread.java:748)
   [junit4]   1>
   [junit4]   1> 12:40:19.858 [qtp714361647-939] ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: Unsupported operation: foobar
   [junit4]   1> at org.apache.solr.handler.admin.CoreAdminHandler.handleCustomAction(CoreAdminHandler.java:220)
   [junit4]   1> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:169)
   [junit4]   1> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
   [junit4]   1> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
   [junit4]   1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   1> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
   [junit4]   1> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   1> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
   [junit4]   1> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   1> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
   [junit4]   1> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   1> at org.eclipse.jetty.server.Server.handle(Server.java:531)
   [junit4]   1> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
   [junit4]   1> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   1> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
   [junit4]   1> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
   [junit4]   1> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
   [junit4]   1> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
   [junit4]   1> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
   [junit4]   1> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
   [junit4]   1> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
   [junit4]   1> at java.lang.Thread.run(Thread.java:748)
   [junit4]   1>
   [junit4]   1> 12:40:28.044 [SUITE-CloudSolrClientTest-seed#[49835F707737C5C]-worker] ERROR org.apache.zookeeper.server.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.impl.CloudSolrClientTest_49835F707737C5C-001
   [junit4]   2> Oct 10, 2018 12:40:28 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {_version_=Lucene50(blocksize=128), multiDefault=PostingsFormat(name=Asserting), a_t=Lucene50(blocksize=128), title_s=PostingsFormat(name=Asserting), intDefault=Lucene50(blocksize=128), id=PostingsFormat(name=Direct), text=PostingsFormat(name=LuceneVarGapFixedInterval), timestamp=Lucene50(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Asserting), multiDefault=DocValuesFormat(name=Memory), a_t=DocValuesFormat(name=Asserting), title_s=DocValuesFormat(name=Memory), intDefault=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Lucene70), text=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=88, maxMBSortInHeap=6.651619911223728, sim=RandomSimilarity(queryNorm=false): {}, locale=fr-CA, timezone=America/Puerto_Rico
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=173075144,total=453509120
   [junit4]   2> NOTE: All tests run in this JVM: [ShardParamsTest, TestRetryUtil, CorrelationEvaluatorTest, TestValidatingJsonMap, TestUpdateRequestCodec, AbsoluteValueEvaluatorTest, FieldAnalysisResponseTest, ConcatOperationTest, SubtractEvaluatorTest, FloorEvaluatorTest, ModuloEvaluatorTest, UniformDistributionEvaluatorTest, SolrSchemalessExampleTest, OrEvaluatorTest, RecursiveEvaluatorTest, LargeVolumeEmbeddedTest, JsonValidatorTest, GraphExpressionTest, CommonAdminParamsTest, TestSpellCheckResponse, SolrDocumentTest, SolrExampleEmbeddedTest, ConcurrentUpdateSolrClientBuilderTest, SolrPingTest, ArcTangentEvaluatorTest, CollectionAdminRequestRequiredParamsTest, GetByIdTest, SolrExampleBinaryTest, SolrExampleXMLTest, TestSolrJErrorHandling, SolrExampleStreamingBinaryTest, TestEmbeddedSolrServer, TestSolrProperties, CloudSolrClientBuilderTest, CloudSolrClientMultiConstructorTest, CloudSolrClientRetryTest, CloudSolrClientTest]
   [junit4] Completed [140/157 (1!)] on J1 in 70.10s, 16 tests, 1 error <<< FAILURES!

[...truncated 44808 lines...]
-ecj-javadoc-lint-tests:
    [mkdir] Created dir: /tmp/ecj374092652
 [ecj-lint] Compiling 932 source files to /tmp/ecj374092652
 [ecj-lint] invalid Class-Path header in manifest of jar file: /x1/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: /x1/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] ----------
 [ecj-lint] 1. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test/org/apache/solr/analysis/TokenizerChainTest.java (at line 37)
 [ecj-lint] TokenizerChain tokenizerChain = new TokenizerChain(
 [ecj-lint]               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'tokenizerChain' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 2. ERROR in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/OverseerCollectionConfigSetProcessorTest.java (at line 81)
 [ecj-lint] import static org.apache.solr.common.params.CollectionAdminParams.COLLECTION;
 [ecj-lint]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.common.params.CollectionAdminParams.COLLECTION is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 3. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/autoscaling/sim/SimCloudManager.java (at line 712)
 [ecj-lint] throw new UnsupportedOperationException("must add at least 1 node first");
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'queryRequest' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 4. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test/org/apache/solr/cloud/autoscaling/sim/SimCloudManager.java (at line 718)
 [ecj-lint] throw new UnsupportedOperationException("must add at least 1 node first");
 [ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'queryRequest' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 5. WARNING in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test/org/apache/solr/update/processor/DistributedUpdateProcessorTest.java (at line 39)
 [ecj-lint] DistributedUpdateProcessor processor = new DistributedUpdateProcessor(
 [ecj-lint]                           ^^^^^^^^^
 [ecj-lint] Resource leak: 'processor' is never closed
 [ecj-lint] ----------
 [ecj-lint] 5 problems (1 error, 4 warnings)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:633: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:101: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build.xml:680: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2092: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2125: Compile failed; see the compiler error output for details.

Total time: 106 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-Tests-7.x - Build # 938 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/938/

9 tests failed.
FAILED:  org.apache.lucene.codecs.lucene60.TestLucene60PointsFormat.testRandomBinaryMedium

Error Message:
max packed value has incorrect length=15 vs expected=12 for docID=-1 field="field"

Stack Trace:
java.lang.RuntimeException: max packed value has incorrect length=15 vs expected=12 for docID=-1 field="field"
        at __randomizedtesting.SeedInfo.seed([F7C415E57CE326E2:80E992A5A84F5335]:0)
        at org.apache.lucene.index.CheckIndex$VerifyPointsVisitor.checkPackedValue(CheckIndex.java:2001)
        at org.apache.lucene.index.CheckIndex$VerifyPointsVisitor.compare(CheckIndex.java:1958)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:746)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
        at org.apache.lucene.index.CheckIndex.testPoints(CheckIndex.java:1810)
        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:735)
        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:862)
        at org.apache.lucene.index.BasePointsFormatTestCase.switchIndex(BasePointsFormatTestCase.java:1002)
        at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:726)
        at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:551)
        at org.apache.lucene.index.BasePointsFormatTestCase.doTestRandomBinary(BasePointsFormatTestCase.java:539)
        at org.apache.lucene.index.BasePointsFormatTestCase.testRandomBinaryMedium(BasePointsFormatTestCase.java:512)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.lang.RuntimeException: max packed value has incorrect length=15 vs expected=12 for docID=-1 field="field"
                at org.apache.lucene.index.CheckIndex$VerifyPointsVisitor.checkPackedValue(CheckIndex.java:2001)
                at org.apache.lucene.index.CheckIndex$VerifyPointsVisitor.compare(CheckIndex.java:1958)
                at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:746)
                at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
                at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
                at org.apache.lucene.index.CheckIndex.testPoints(CheckIndex.java:1810)
                at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:735)
                at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
                at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:862)
                at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:559)
                ... 38 more


FAILED:  org.apache.lucene.util.bkd.TestBKD.testOneDimEqual

Error Message:
bytesPerDim=28 splitDim=1 numIndexDims=2 numDataDims=3

Stack Trace:
java.lang.AssertionError: bytesPerDim=28 splitDim=1 numIndexDims=2 numDataDims=3
        at __randomizedtesting.SeedInfo.seed([F7C415E57CE326E2:495A713304621B96]:0)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:781)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
        at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:765)
        at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:628)
        at org.apache.lucene.util.bkd.TestBKD.testOneDimEqual(TestBKD.java:517)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.lucene.util.bkd.TestBKD.testOneDimLowCard

Error Message:
bytesPerDim=9 splitDim=1 numIndexDims=2 numDataDims=4

Stack Trace:
java.lang.AssertionError: bytesPerDim=9 splitDim=1 numIndexDims=2 numDataDims=4
        at __randomizedtesting.SeedInfo.seed([F7C415E57CE326E2:FDEFF36D3404C97B]:0)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:781)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
        at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:765)
        at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:628)
        at org.apache.lucene.util.bkd.TestBKD.testOneDimLowCard(TestBKD.java:555)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.lucene.util.bkd.TestBKD.testMultiValued

Error Message:
bytesPerDim=6 splitDim=0 numIndexDims=1 numDataDims=5

Stack Trace:
java.lang.AssertionError: bytesPerDim=6 splitDim=0 numIndexDims=1 numDataDims=5
        at __randomizedtesting.SeedInfo.seed([F7C415E57CE326E2:23E471D7B22166AA]:0)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:781)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
        at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:765)
        at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:628)
        at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:621)
        at org.apache.lucene.util.bkd.TestBKD.testMultiValued(TestBKD.java:616)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.lucene.codecs.asserting.TestAssertingPointsFormat.testRandomBinaryMedium

Error Message:
bytesPerDim=11 splitDim=0 numIndexDims=1 numDataDims=2

Stack Trace:
java.lang.AssertionError: bytesPerDim=11 splitDim=0 numIndexDims=1 numDataDims=2
        at __randomizedtesting.SeedInfo.seed([C29D6D4442C7BF51:B5B0EA04966BCA86]:0)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:781)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
        at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
        at org.apache.lucene.index.AssertingLeafReader$AssertingPointValues.intersect(AssertingLeafReader.java:898)
        at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:832)
        at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:551)
        at org.apache.lucene.index.BasePointsFormatTestCase.doTestRandomBinary(BasePointsFormatTestCase.java:539)
        at org.apache.lucene.index.BasePointsFormatTestCase.testRandomBinaryMedium(BasePointsFormatTestCase.java:512)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.lang.AssertionError: bytesPerDim=11 splitDim=0 numIndexDims=1 numDataDims=2
                at org.apache.lucene.util.bkd.BKDReader.estimatePointCount(BKDReader.java:835)
                at org.apache.lucene.util.bkd.BKDReader.estimatePointCount(BKDReader.java:841)
                at org.apache.lucene.util.bkd.BKDReader.estimatePointCount(BKDReader.java:538)
                at org.apache.lucene.index.AssertingLeafReader$AssertingPointValues.estimatePointCount(AssertingLeafReader.java:903)
                at org.apache.lucene.index.CheckIndex.testPoints(CheckIndex.java:1796)
                at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:735)
                at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
                at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:862)
                at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:559)
                ... 38 more


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([81F4A36F3D37DD32:9A09CB593CBB0CA]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.MultiThreadedOCPTest.testFillWorkQueue(MultiThreadedOCPTest.java:114)
        at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestPullReplicaErrorHandling.testPullReplicaDisconnectsFromZooKeeper

Error Message:
IOException occured when talking to server at: http://127.0.0.1:35189/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:35189/solr
        at __randomizedtesting.SeedInfo.seed([81F4A36F3D37DD32:9EBD83EC18DDD252]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.TestPullReplicaErrorHandling.tearDown(TestPullReplicaErrorHandling.java:128)
        at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:993)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:35189 failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 46 more


FAILED:  org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToPullReplica

Error Message:
IOException occured when talking to server at: http://127.0.0.1:35189/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:35189/solr
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToPullReplica(TestPullReplicaErrorHandling.java:142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:35189 failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 49 more


FAILED:  org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader

Error Message:
IOException occured when talking to server at: http://127.0.0.1:35189/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:35189/solr
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader(TestPullReplicaErrorHandling.java:183)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:35189 failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 49 more




Build Log:
[...truncated 828 lines...]
   [junit4] Suite: org.apache.lucene.util.bkd.TestBKD
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBKD -Dtests.method=testOneDimEqual -Dtests.seed=F7C415E57CE326E2 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sk -Dtests.timezone=Africa/Porto-Novo -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.06s J1 | TestBKD.testOneDimEqual <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: bytesPerDim=28 splitDim=1 numIndexDims=2 numDataDims=3
   [junit4]    > at __randomizedtesting.SeedInfo.seed([F7C415E57CE326E2:495A713304621B96]:0)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:781)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
   [junit4]    > at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:765)
   [junit4]    > at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:628)
   [junit4]    > at org.apache.lucene.util.bkd.TestBKD.testOneDimEqual(TestBKD.java:517)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBKD -Dtests.method=testOneDimLowCard -Dtests.seed=F7C415E57CE326E2 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sk -Dtests.timezone=Africa/Porto-Novo -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.59s J1 | TestBKD.testOneDimLowCard <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: bytesPerDim=9 splitDim=1 numIndexDims=2 numDataDims=4
   [junit4]    > at __randomizedtesting.SeedInfo.seed([F7C415E57CE326E2:FDEFF36D3404C97B]:0)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:781)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
   [junit4]    > at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:765)
   [junit4]    > at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:628)
   [junit4]    > at org.apache.lucene.util.bkd.TestBKD.testOneDimLowCard(TestBKD.java:555)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBKD -Dtests.method=testMultiValued -Dtests.seed=F7C415E57CE326E2 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sk -Dtests.timezone=Africa/Porto-Novo -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.12s J1 | TestBKD.testMultiValued <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: bytesPerDim=6 splitDim=0 numIndexDims=1 numDataDims=5
   [junit4]    > at __randomizedtesting.SeedInfo.seed([F7C415E57CE326E2:23E471D7B22166AA]:0)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:781)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
   [junit4]    > at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:765)
   [junit4]    > at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:628)
   [junit4]    > at org.apache.lucene.util.bkd.TestBKD.verify(TestBKD.java:621)
   [junit4]    > at org.apache.lucene.util.bkd.TestBKD.testMultiValued(TestBKD.java:616)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4] IGNOR/A 0.00s J1 | TestBKD.testRandomBinaryBig
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/J1/temp/lucene.util.bkd.TestBKD_F7C415E57CE326E2-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1213, maxMBSortInHeap=5.67013175349037, sim=RandomSimilarity(queryNorm=true): {}, locale=sk, timezone=Africa/Porto-Novo
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=125396056,total=364380160
   [junit4]   2> NOTE: All tests run in this JVM: [TestEarlyTerminatingSortingCollector, TestSegmentCacheables, TestDocValuesScoring, TestConsistentFieldNumbers, TestIndexWriterThreadsToSegments, TestMultiset, TestSoftDeletesDirectoryReaderWrapper, TestFeatureField, TestGraphTokenStreamFiniteStrings, TestSegmentMerger, TestStressIndexing2, TestMultiCollector, TestIntsRef, TestPolygon2D, TestIndexWriterOnVMError, TestSimilarityProvider, TestBytesRefHash, TestCustomNorms, TestSimpleExplanationsWithFillerDocs, TestParallelCompositeReader, TestIntBlockPool, TestSearchAfter, TestSortRescorer, TestSpanFirstQuery, TestPriorityQueue, TestPhraseQuery, TestMmapDirectory, TestBoolean2ScorerSupplier, TestRateLimiter, TestSearch, TestCharacterUtils, TestFieldCacheRewriteMethod, TestBasics, TestOmitTf, TestIndexWriterCommit, TestSmallFloat, TestDocValuesIndexing, Test2BPagedBytes, TestDateTools, TestNumericUtils, TestBagOfPositions, TestFastDecompressionMode, TestNumericDocValuesUpdates, TestIndexInput, TestLucene70NormsFormat, TestLucene50StoredFieldsFormatHighCompression, TestIndexWriterForceMerge, TestStringHelper, TestByteBlockPool, TestGeoEncodingUtils, TestNearSpansOrdered, TestIndexSearcher, TestSpanTermQuery, Test2BNumericDocValues, TestCustomSearcherSort, TestFilterIterator, TestReusableStringReader, TestPositiveScoresOnlyCollector, TestSwappedIndexFiles, TestBKD]
   [junit4] Completed [162/486 (1!)] on J1 in 17.19s, 19 tests, 3 failures, 1 skipped <<< FAILURES!

[...truncated 207 lines...]
   [junit4] Suite: org.apache.lucene.codecs.lucene60.TestLucene60PointsFormat
   [junit4] IGNOR/A 0.00s J2 | TestLucene60PointsFormat.testRandomBinaryBig
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4] IGNOR/A 0.02s J2 | TestLucene60PointsFormat.testMergeStability
   [junit4]    > Assumption #1: TODO: mess with the parameters and test gets angry!
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLucene60PointsFormat -Dtests.method=testRandomBinaryMedium -Dtests.seed=F7C415E57CE326E2 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=en-PH -Dtests.timezone=Asia/Magadan -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.66s J2 | TestLucene60PointsFormat.testRandomBinaryMedium <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: max packed value has incorrect length=15 vs expected=12 for docID=-1 field="field"
   [junit4]    > at __randomizedtesting.SeedInfo.seed([F7C415E57CE326E2:80E992A5A84F5335]:0)
   [junit4]    > at org.apache.lucene.index.CheckIndex$VerifyPointsVisitor.checkPackedValue(CheckIndex.java:2001)
   [junit4]    > at org.apache.lucene.index.CheckIndex$VerifyPointsVisitor.compare(CheckIndex.java:1958)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:746)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
   [junit4]    > at org.apache.lucene.index.CheckIndex.testPoints(CheckIndex.java:1810)
   [junit4]    > at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:735)
   [junit4]    > at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
   [junit4]    > at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:862)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.switchIndex(BasePointsFormatTestCase.java:1002)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:726)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:551)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.doTestRandomBinary(BasePointsFormatTestCase.java:539)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.testRandomBinaryMedium(BasePointsFormatTestCase.java:512)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Suppressed: java.lang.RuntimeException: max packed value has incorrect length=15 vs expected=12 for docID=-1 field="field"
   [junit4]    > at org.apache.lucene.index.CheckIndex$VerifyPointsVisitor.checkPackedValue(CheckIndex.java:2001)
   [junit4]    > at org.apache.lucene.index.CheckIndex$VerifyPointsVisitor.compare(CheckIndex.java:1958)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:746)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
   [junit4]    > at org.apache.lucene.index.CheckIndex.testPoints(CheckIndex.java:1810)
   [junit4]    > at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:735)
   [junit4]    > at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
   [junit4]    > at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:862)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:559)
   [junit4]    > ... 38 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/J2/temp/lucene.codecs.lucene60.TestLucene60PointsFormat_F7C415E57CE326E2-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1168, maxMBSortInHeap=6.274619012255403, sim=RandomSimilarity(queryNorm=false): {}, locale=en-PH, timezone=Asia/Magadan
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=239712944,total=425197568
   [junit4]   2> NOTE: All tests run in this JVM: [TestEarlyTermination, TestShardSearching, TestDoubleRangeFieldQueries, TestCrash, TestIndexWriterUnicode, TestNGramPhraseQuery, TestSoftDeletesRetentionMergePolicy, TestField, TestMutableValues, TestSegmentReader, TestStressNRT, TestMultiPhraseEnum, TestLSBRadixSorter, Test2BBinaryDocValues, TestIndexWriterOutOfFileDescriptors, TestSimpleExplanations, TestCharsRef, TestCustomTermFreq, TestSimpleSearchEquivalence, TestParallelLeafReader, TestIsCurrent, TestSearchWithThreads, TestSortedNumericSortField, TestSpanMultiTermQueryWrapper, TestQueryBuilder, TestPointQueries, TestMultiMMap, TestBooleanMinShouldMatch, TestSimpleFSDirectory, TestSearchForDuplicates, TestDelegatingAnalyzerWrapper, TestFieldValueQuery, TestFieldMaskingSpanQuery, TestOneMergeWrappingMergePolicy, TestIndexWriterConfig, TestSparseFixedBitDocIdSet, TestDocsAndPositions, TestArrayUtil, TestDocument, TestOfflineSorter, TestBagOfPostings, TestHighCompressionMode, TestOmitNorms, TestIndexManyDocuments, TestLucene70SegmentInfoFormat, TestLucene50TermVectorsFormat, TestIndexWriterFromReader, TestStringMSBRadixSorter, TestBytesRef, TestGeoUtils, TestSpanBoostQuery, TestIntRangeFieldQueries, TestSpans, Test2BPoints, TestDateSort, TestFixedBitDocIdSet, TestStopFilter, TestPrefixInBooleanQuery, TestTerm, TestDocIdsWriter, TestFieldsReader, TestWeakIdentityMap, TestDocValues, TestDocCount, TestTermVectorsReader, TestFieldInfos, TestAllFilesHaveCodecHeader, TestLongValuesSource, TestFuzzyQuery, TestBoostQuery, TestDocumentsWriterDeleteQueue, TestComplexExplanationsOfNonMatches, TestBufferedIndexInput, TestSleepingLockWrapper, Test2BPostingsBytes, TestLucene50LiveDocsFormat, TestNoMergeScheduler, TestAllFilesCheckIndexHeader, TestWildcard, TestLatLonPointDistanceSort, TestBytesRefAttImpl, TestCharTermAttributeImpl, TestPackedTokenAttributeImpl, TestSimpleAttributeImpl, TestCodecUtil, TestBlockPostingsFormat, TestLucene60FieldInfoFormat, TestLucene60PointsFormat]
   [junit4] Completed [227/486 (2!)] on J2 in 21.14s, 20 tests, 1 error, 2 skipped <<< FAILURES!

[...truncated 1132 lines...]
   [junit4] Suite: org.apache.lucene.codecs.asserting.TestAssertingPointsFormat
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestAssertingPointsFormat -Dtests.method=testRandomBinaryMedium -Dtests.seed=C29D6D4442C7BF51 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=en-GB -Dtests.timezone=Asia/Yerevan -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.45s J0 | TestAssertingPointsFormat.testRandomBinaryMedium <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: bytesPerDim=11 splitDim=0 numIndexDims=1 numDataDims=2
   [junit4]    > at __randomizedtesting.SeedInfo.seed([C29D6D4442C7BF51:B5B0EA04966BCA86]:0)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:781)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:787)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:533)
   [junit4]    > at org.apache.lucene.index.AssertingLeafReader$AssertingPointValues.intersect(AssertingLeafReader.java:898)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:832)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:551)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.doTestRandomBinary(BasePointsFormatTestCase.java:539)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.testRandomBinaryMedium(BasePointsFormatTestCase.java:512)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Suppressed: java.lang.AssertionError: bytesPerDim=11 splitDim=0 numIndexDims=1 numDataDims=2
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.estimatePointCount(BKDReader.java:835)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.estimatePointCount(BKDReader.java:841)
   [junit4]    > at org.apache.lucene.util.bkd.BKDReader.estimatePointCount(BKDReader.java:538)
   [junit4]    > at org.apache.lucene.index.AssertingLeafReader$AssertingPointValues.estimatePointCount(AssertingLeafReader.java:903)
   [junit4]    > at org.apache.lucene.index.CheckIndex.testPoints(CheckIndex.java:1796)
   [junit4]    > at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:735)
   [junit4]    > at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:300)
   [junit4]    > at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:862)
   [junit4]    > at org.apache.lucene.index.BasePointsFormatTestCase.verify(BasePointsFormatTestCase.java:559)
   [junit4]    > ... 38 more
   [junit4] IGNOR/A 0.00s J0 | TestAssertingPointsFormat.testMergeStability
   [junit4]    > Assumption #1: merge is not stable
   [junit4] IGNOR/A 0.00s J0 | TestAssertingPointsFormat.testRandomBinaryBig
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly())
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/test-framework/test/J0/temp/lucene.codecs.asserting.TestAssertingPointsFormat_C29D6D4442C7BF51-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=en-GB, timezone=Asia/Yerevan
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=209657384,total=341835776
   [junit4]   2> NOTE: All tests run in this JVM: [TestBaseExplanationTestCase, TestCompressingTermVectorsFormat, TestLookaheadTokenFilter, TestHandleTrackingFS, Nested1, Nested1, NestedSetupChain, NestedTeardownChain, TestAssertingPointsFormat]
   [junit4] Completed [23/41 (1!)] on J0 in 16.59s, 18 tests, 1 failure, 2 skipped <<< FAILURES!

[...truncated 11859 lines...]
   [junit4] Suite: org.apache.solr.cloud.MultiThreadedOCPTest
   [junit4]   2> 2959433 INFO  (SUITE-MultiThreadedOCPTest-seed#[81F4A36F3D37DD32]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest_81F4A36F3D37DD32-001/init-core-data-001
   [junit4]   2> 2959434 WARN  (SUITE-MultiThreadedOCPTest-seed#[81F4A36F3D37DD32]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 2959434 INFO  (SUITE-MultiThreadedOCPTest-seed#[81F4A36F3D37DD32]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2959525 INFO  (SUITE-MultiThreadedOCPTest-seed#[81F4A36F3D37DD32]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2959525 INFO  (SUITE-MultiThreadedOCPTest-seed#[81F4A36F3D37DD32]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /d_bdb/i
   [junit4]   2> 2959618 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2959655 INFO  (Thread-3827) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2959655 INFO  (Thread-3827) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2959706 ERROR (Thread-3827) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2959744 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.ZkTestServer start zk server on port:35328
   [junit4]   2> 2959941 INFO  (zkConnectionManagerCallback-3739-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2960075 INFO  (zkConnectionManagerCallback-3741-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2960079 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2960092 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2960094 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2960094 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2960095 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2960129 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2960129 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2960130 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2960131 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2960148 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2960149 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2960185 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2960805 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 2960875 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2960875 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2960875 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2960877 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@49e27329{/d_bdb/i,null,AVAILABLE}
   [junit4]   2> 2960878 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2d1b510b{HTTP/1.1,[http/1.1]}{127.0.0.1:41762}
   [junit4]   2> 2960878 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.e.j.s.Server Started @2961143ms
   [junit4]   2> 2960878 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest_81F4A36F3D37DD32-001/tempDir-001/control/data, hostContext=/d_bdb/i, hostPort=41762, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest_81F4A36F3D37DD32-001/control-001/cores}
   [junit4]   2> 2960878 ERROR (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2960878 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2960878 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 2960878 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2960878 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2960879 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-10-10T18:28:00.040Z
   [junit4]   2> 2960915 INFO  (zkConnectionManagerCallback-3743-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2960941 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2960941 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest_81F4A36F3D37DD32-001/control-001/solr.xml
   [junit4]   2> 2960977 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2960977 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2960978 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@357df1b1, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2962142 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35328/solr
   [junit4]   2> 2962214 INFO  (zkConnectionManagerCallback-3747-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2962255 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x105037ed3680003, likely client has closed socket
   [junit4]   2> 2962308 INFO  (zkConnectionManagerCallback-3749-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2963564 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41762_d_bdb%2Fi
   [junit4]   2> 2963564 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.c.Overseer Overseer (id=73468812169052164-127.0.0.1:41762_d_bdb%2Fi-n_0000000000) starting
   [junit4]   2> 2963607 INFO  (zkConnectionManagerCallback-3756-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2963682 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35328/solr ready
   [junit4]   2> 2963693 INFO  (OverseerStateUpdate-73468812169052164-127.0.0.1:41762_d_bdb%2Fi-n_0000000000) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:41762_d_bdb%2Fi
   [junit4]   2> 2963710 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41762_d_bdb%2Fi
   [junit4]   2> 2963740 INFO  (zkCallback-3748-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2963740 INFO  (zkCallback-3755-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2963764 DEBUG (OverseerAutoScalingTriggerThread-73468812169052164-127.0.0.1:41762_d_bdb%2Fi-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:41762_d_bdb%2Fi]
   [junit4]   2> 2963777 DEBUG (ScheduledTrigger-7392-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 2963862 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2963931 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@357df1b1
   [junit4]   2> 2963982 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@357df1b1
   [junit4]   2> 2963983 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@357df1b1
   [junit4]   2> 2963996 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest_81F4A36F3D37DD32-001/control-001/cores
   [junit4]   2> 2964110 INFO  (zkConnectionManagerCallback-3762-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2964250 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2964251 INFO  (TEST-MultiThreadedOCPTest.test-seed#[81F4A36F3D37DD32]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35328/solr ready
   [junit4]   2> 2964305 INFO  (qtp532509718-16856) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:41762_d_bdb%252Fi&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2964430 INFO  (OverseerThreadFactory-7394-thread-1-processing-n:127.0.0.1:41762_d_bdb%2Fi) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2964451 DEBUG (OverseerStateUpdate-73468812169052164-127.0.0.1:41762_d_bdb%2Fi-n_0000000000) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"control_collection",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "createNodeSet":"127.0.0.1:41762_d_bdb%2Fi",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 2964553 INFO  (qtp532509718-16851) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2964569 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2964587 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2964694 DEBUG (OverseerStateUpdate-73468812169052164-127.0.0.1:41762_d_bdb%2Fi-n_0000000000) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41762/d_bdb/i",
   [junit4]   2>   "node_name":"127.0.0.1:41762_d_bdb%2Fi",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2964842 DEBUG (ScheduledTrigger-7392-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 2965825 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 2965842 DEBUG (ScheduledTrigger-7392-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 2965883 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2966592 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2966697 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2966698 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@357df1b1
   [junit4]   2> 2966715 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest_81F4A36F3D37DD32-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest_81F4A36F3D37DD32-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2966717 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=46.1416015625, floorSegmentMB=0.8251953125, forceMergeDeletesPctAllowed=8.21867821042121, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5503763364214344, deletesPctAllowed=24.15783404536201
   [junit4]   2> 2966748 WARN  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2966848 DEBUG (ScheduledTrigger-7392-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 2967307 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2967307 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2967309 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2967309 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2967310 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6555990350618188]
   [junit4]   2> 2967310 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3fa0b95f[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2967328 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2967328 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2967328 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2967329 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613964075587338240
   [junit4]   2> 2967345 INFO  (searcherExecutor-7399-thread-1-processing-n:127.0.0.1:41762_d_bdb%2Fi x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@3fa0b95f[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2967347 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 2967382 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2967382 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2967382 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41762/d_bdb/i/control_collection_shard1_replica_n1/
   [junit4]   2> 2967382 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2967382 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:41762/d_bdb/i/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2967382 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2967402 INFO  (qtp532509718-16852) [n:127.0.0.1:41762_d_bdb%2Fi c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41762/d_bdb/i/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2967445 DEBUG (OverseerStateUpdate-73468812169052164-127.0.0.1:41762_d_bdb%2Fi-n_0000000000) [n:127.0.0.1:41762_d_bdb%2Fi    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:41762/d_bdb/i",
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "state":"active"} current state version: 0
   [junit4]   2> 2967510 INFO  (q

[...truncated too long message...]

operty disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/null317540543
     [copy] Copying 239 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/null317540543
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/null317540543

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 59ms :: artifacts dl 6ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 215 minutes 53 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]