[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1447 - Failure

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1447 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1447/

7 tests failed.
FAILED:  org.apache.lucene.analysis.core.TestRandomChains.testRandomChains

Error Message:
startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=0,endOffset=11,lastStartOffset=7 for field 'dummy'

Stack Trace:
java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=0,endOffset=11,lastStartOffset=7 for field 'dummy'
        at __randomizedtesting.SeedInfo.seed([7CA2061763145E4A:41432F762406438A]:0)
        at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:767)
        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:392)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:240)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:497)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1727)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1462)
        at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
        at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:672)
        at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:562)
        at org.apache.lucene.analysis.core.TestRandomChains.testRandomChains(TestRandomChains.java:856)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([D0F329E6A211B5A0:722B673CC6E39CA6]:0)
        at org.apache.lucene.util.packed.Packed8ThreeBlocks.<init>(Packed8ThreeBlocks.java:41)
        at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:963)
        at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:939)
        at org.apache.lucene.util.packed.GrowableWriter.ensureCapacity(GrowableWriter.java:80)
        at org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:88)
        at org.apache.lucene.util.packed.AbstractPagedMutable.set(AbstractPagedMutable.java:98)
        at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:152)
        at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
        at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
        at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
        at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:296)
        at org.apache.lucene.util.fst.Builder.add(Builder.java:400)
        at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.writeFST(MemoryDocValuesConsumer.java:373)
        at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:416)
        at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:406)
        at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:527)
        at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:139)
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
        at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:181)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:125)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4441)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4081)
        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2245)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5096)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1730)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1462)
        at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2294)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
        at __randomizedtesting.SeedInfo.seed([CFE48D7EA6F46476:651CFD0AF93A283]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue(TriggerIntegrationTest.java:711)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:10010_solr should have been deleted

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10010_solr should have been deleted
        at __randomizedtesting.SeedInfo.seed([CFE48D7EA6F46476:D75E0572A8C1A999]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertFalse(Assert.java:68)
        at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration(TestTriggerIntegration.java:842)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsBasicDistributedZk2Test.test

Error Message:
.response.numFound:36!=67

Stack Trace:
junit.framework.AssertionFailedError: .response.numFound:36!=67
        at __randomizedtesting.SeedInfo.seed([CFE48D7EA6F46476:47B0B2A40808098E]:0)
        at junit.framework.Assert.fail(Assert.java:50)
        at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:913)
        at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:940)
        at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:607)
        at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:587)
        at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:566)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkQueries(AbstractFullDistribZkTestBase.java:942)
        at org.apache.solr.cloud.BasicDistributedZk2Test.test(BasicDistributedZk2Test.java:102)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest.test

Error Message:
Mismatch in counts between replicas

Stack Trace:
java.lang.AssertionError: Mismatch in counts between replicas
        at __randomizedtesting.SeedInfo.seed([CFE48D7EA6F46476:47B0B2A40808098E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.RecoveryZkTest.assertShardConsistency(RecoveryZkTest.java:143)
        at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:126)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest.testParse

Error Message:
Error from server at https://127.0.0.1:38512/solr: Collection : myalias_2017-10-24 is part of alias myalias remove or modify the alias before removing this collection.

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:38512/solr: Collection : myalias_2017-10-24 is part of alias myalias remove or modify the alias before removing this collection.
        at __randomizedtesting.SeedInfo.seed([CFE48D7EA6F46476:A061C6261A825799]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest.doBefore(TimeRoutedAliasUpdateProcessorTest.java:84)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1772 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexSorting
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexSorting -Dtests.method=testRandom3 -Dtests.seed=D0F329E6A211B5A0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-GT -Dtests.timezone=Asia/Tokyo -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   1437s J0 | TestIndexSorting.testRandom3 <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([D0F329E6A211B5A0:722B673CC6E39CA6]:0)
   [junit4]    > at org.apache.lucene.util.packed.Packed8ThreeBlocks.<init>(Packed8ThreeBlocks.java:41)
   [junit4]    > at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:963)
   [junit4]    > at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:939)
   [junit4]    > at org.apache.lucene.util.packed.GrowableWriter.ensureCapacity(GrowableWriter.java:80)
   [junit4]    > at org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:88)
   [junit4]    > at org.apache.lucene.util.packed.AbstractPagedMutable.set(AbstractPagedMutable.java:98)
   [junit4]    > at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:152)
   [junit4]    > at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
   [junit4]    > at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
   [junit4]    > at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
   [junit4]    > at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:296)
   [junit4]    > at org.apache.lucene.util.fst.Builder.add(Builder.java:400)
   [junit4]    > at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.writeFST(MemoryDocValuesConsumer.java:373)
   [junit4]    > at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:416)
   [junit4]    > at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:406)
   [junit4]    > at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:527)
   [junit4]    > at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:139)
   [junit4]    > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:181)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:125)
   [junit4]    > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4441)
   [junit4]    > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4081)
   [junit4]    > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]    > at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2245)
   [junit4]    > at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5096)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1730)
   [junit4]    > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1462)
   [junit4]    > at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2294)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J0/temp/lucene.index.TestIndexSorting_D0F329E6A211B5A0-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {sparse_text=FSTOrd50, docs=BlockTreeOrds(blocksize=128), norms=BlockTreeOrds(blocksize=128), positions=FSTOrd50, id=BlockTreeOrds(blocksize=128), term_vectors=PostingsFormat(name=Memory)}, docValues:{multi_valued_long=DocValuesFormat(name=Lucene70), foo=DocValuesFormat(name=Lucene70), numeric=DocValuesFormat(name=Direct), multi_valued_numeric=DocValuesFormat(name=Memory), float=DocValuesFormat(name=Asserting), long=DocValuesFormat(name=Asserting), points=DocValuesFormat(name=Memory), sparse_int=DocValuesFormat(name=Asserting), bar=DocValuesFormat(name=Memory), multi_valued_double=DocValuesFormat(name=Memory), docs=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Memory), multi_valued_int=DocValuesFormat(name=Direct), multi_valued_bytes=DocValuesFormat(name=Direct), sparse_text=DocValuesFormat(name=Lucene70), double=DocValuesFormat(name=Direct), sparse_binary=DocValuesFormat(name=Lucene70), positions=DocValuesFormat(name=Lucene70), int=DocValuesFormat(name=Memory), sorted=DocValuesFormat(name=Direct), sparse=DocValuesFormat(name=Lucene70), multi_valued_string=DocValuesFormat(name=Memory), norms=DocValuesFormat(name=Memory), bytes=DocValuesFormat(name=Memory), dense_int=DocValuesFormat(name=Memory), binary=DocValuesFormat(name=Direct), multi_valued_float=DocValuesFormat(name=Asserting), term_vectors=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1742, maxMBSortInHeap=7.990190520330529, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5db79bb1), locale=es-GT, timezone=Asia/Tokyo
   [junit4]   2> NOTE: Linux 4.4.0-104-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=341628296,total=374865920
   [junit4]   2> NOTE: All tests run in this JVM: [TestSortRescorer, TestSpanContainQuery, TestPrefixRandom, TestAttributeSource, TestIntroSorter, TestIndexInput, TestStandardAnalyzer, TestIndexWriterThreadsToSegments, TestDocumentWriter, TestIndexWriter, TestInPlaceMergeSorter, TestNRTCachingDirectory, TestBoolean2, TestCachingCollector, TestMaxTermFrequency, TestSpanFirstQuery, TestForceMergeForever, TestCodecs, TestDocsAndPositions, TestDemo, TestBytesRefArray, TestOmitNorms, Test2BNumericDocValues, TestMultiThreadTermVectors, TestSimpleExplanations, TestTermVectors, TestMinShouldMatch2, TestDeterminizeLexicon, TestMSBRadixSorter, TestPointQueries, TestBufferedChecksum, TestRegExp, TestPerFieldPostingsFormat, TestSearchAfter, TestRollingBuffer, TestFilterLeafReader, TestIndexedDISI, TestSloppyPhraseQuery, TestDocIDMerger, Test2BPoints, TestBasicModelIF, TestLucene70SegmentInfoFormat, TestNot, TestDocIdsWriter, TestBinaryTerms, TestMultiLevelSkipList, TestBooleanSimilarity, TestMultiset, TestTimSorter, TestBinaryDocValuesUpdates, TestRateLimiter, TestFilterCodecReader, TestIndexWriterWithThreads, TestNotDocIdSet, TestParallelTermEnum, TestExitableDirectoryReader, TestPostingsOffsets, TestSort, TestSimpleAttributeImpl, TestGeoUtils, TestBagOfPositions, TestPhrasePrefixQuery, TestAxiomaticF2LOG, TestAxiomaticF1EXP, TestTopFieldCollectorEarlyTermination, TestPositiveScoresOnlyCollector, TestTermsEnum, TestTwoPhaseCommitTool, TestCrashCausesCorruptIndex, TestIndexWriterForceMerge, TestConstantScoreQuery, TestFloatRangeFieldQueries, TestLogMergePolicy, TestNoDeletionPolicy, TestNoMergeScheduler, TestDistributionSPL, TestSizeBoundedForceMerge, TestCharArrayMap, TestCodecHoldsOpenFiles, TestBytesStore, TestGraphTokenizers, TestAddIndexes, TestShardSearching, TestIndexWriterExceptions, TestLucene70NormsFormat, TestGeoEncodingUtils, Test2BSortedDocValuesFixedSorted, TestAllFilesCheckIndexHeader, TestAllFilesHaveChecksumFooter, TestCustomTermFreq, TestDemoParallelLeafReader, TestExceedMaxTermLength, TestFieldInvertState, TestFieldReuse, TestIndexManyDocuments, TestIndexReaderClose, TestIndexSorting]
   [junit4] Completed [401/476 (1!)] on J0 in 1499.22s, 50 tests, 1 error <<< FAILURES!

[...truncated 275 lines...]
   [junit4] JVM J0: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20180112_101728_5189025759997471894717.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid25196.hprof ...
   [junit4] Heap dump file created [406803273 bytes in 3.977 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 1817 lines...]
   [junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
   [junit4]   2> TEST FAIL: useCharFilter=false text='\u11ef\u1150\u11bb\u1135\u1175\u11f3\u11ac\u1101\u117b g'
   [junit4]   2> Exception from random analyzer:
   [junit4]   2> charfilters=
   [junit4]   2>   org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(java.io.StringReader@3e344857, [<HIRAGANA>])
   [junit4]   2>   org.apache.lucene.analysis.MockCharFilter(org.apache.lucene.analysis.charfilter.HTMLStripCharFilter@18125357)
   [junit4]   2> tokenizer=
   [junit4]   2>   org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer()
   [junit4]   2> filters=
   [junit4]   2>   org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@5c323058 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1)
   [junit4]   2>   org.apache.lucene.analysis.cjk.CJKBigramFilter(ValidatingTokenFilter@68b354ed term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1)
   [junit4]   2>   org.apache.lucene.analysis.sr.SerbianNormalizationFilter(ValidatingTokenFilter@2064359a term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1)
   [junit4]   2>   org.apache.lucene.analysis.util.ElisionFilter(ValidatingTokenFilter@39806abf term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1, [r, utxaqki, isy])
   [junit4]   2> offsetsAreCorrect=false
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomChains -Dtests.method=testRandomChains -Dtests.seed=7CA2061763145E4A -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-CO -Dtests.timezone=Pacific/Pago_Pago -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    340s J2 | TestRandomChains.testRandomChains <<<
   [junit4]    > Throwable #1: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=0,endOffset=11,lastStartOffset=7 for field 'dummy'
   [junit4]    > at __randomizedtesting.SeedInfo.seed([7CA2061763145E4A:41432F762406438A]:0)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:767)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
   [junit4]    > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:392)
   [junit4]    > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:240)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:497)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1727)
   [junit4]    > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1462)
   [junit4]    > at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
   [junit4]    > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:672)
   [junit4]    > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:562)
   [junit4]    > at org.apache.lucene.analysis.core.TestRandomChains.testRandomChains(TestRandomChains.java:856)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/analysis/common/test/J2/temp/lucene.analysis.core.TestRandomChains_7CA2061763145E4A-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@463b128), locale=es-CO, timezone=Pacific/Pago_Pago
   [junit4]   2> NOTE: Linux 4.4.0-104-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=198254528,total=460849152
   [junit4]   2> NOTE: All tests run in this JVM: [TestCJKWidthFilterFactory, TestUAX29URLEmailTokenizer, TestTeeSinkTokenFilter, TestHindiAnalyzer, TestGermanStemFilter, TestLimitTokenPositionFilter, TestSpanishLightStemFilter, TestConv, TestRussianLightStemFilterFactory, TestKStemmer, TestDoubleEscape, TestWordnetSynonymParser, TestCodepointCountFilterFactory, TestNorwegianMinimalStemFilterFactory, TestFrenchLightStemFilter, TestTwoFold, TestSerbianNormalizationFilterFactory, TestSynonymFilterFactory, TypeAsPayloadTokenFilterTest, TestCharTokenizers, NumericPayloadTokenFilterTest, EdgeNGramTokenizerTest, TestPatternReplaceCharFilter, TestLengthFilterFactory, TestArabicFilters, TestScandinavianFoldingFilterFactory, TestCzechStemmer, TestPatternReplaceFilter, TestPersianNormalizationFilter, TestFrenchLightStemFilterFactory, TestPersianNormalizationFilterFactory, TestTrimFilterFactory, GreekAnalyzerTest, TestCharArrayIterator, TestStandardFactories, TestRandomChains]
   [junit4] Completed [273/285 (1!)] on J2 in 597.28s, 2 tests, 1 error <<< FAILURES!

[...truncated 8699 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_CFE48D7EA6F46476-001/init-core-data-001
   [junit4]   2> 585396 WARN  (SUITE-TestTriggerIntegration-seed#[CFE48D7EA6F46476]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 585396 INFO  (SUITE-TestTriggerIntegration-seed#[CFE48D7EA6F46476]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 585397 INFO  (SUITE-TestTriggerIntegration-seed#[CFE48D7EA6F46476]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 585397 INFO  (SUITE-TestTriggerIntegration-seed#[CFE48D7EA6F46476]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 585455 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 585467 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 585467 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 585467 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 585467 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 585480 DEBUG (ScheduledTrigger-681-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585480 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 585480 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 585589 DEBUG (ScheduledTrigger-681-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585637 DEBUG (ScheduledTrigger-681-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585654 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSearchRate
   [junit4]   2> 585654 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 585655 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 585655 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 585657 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 585657 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 585657 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 585657 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 585657 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 585657 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 585657 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 585665 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 585657 DEBUG (ScheduledTrigger-684-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585686 DEBUG (ScheduledTrigger-684-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585723 DEBUG (ScheduledTrigger-684-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585771 DEBUG (ScheduledTrigger-684-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585791 DEBUG (ScheduledTrigger-684-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585811 DEBUG (ScheduledTrigger-684-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585852 DEBUG (ScheduledTrigger-684-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585872 DEBUG (ScheduledTrigger-684-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585893 DEBUG (ScheduledTrigger-684-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585911 DEBUG (simCloudManagerPool-683-thread-4) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 585911 DEBUG (simCloudManagerPool-683-thread-4) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 585925 DEBUG (ScheduledTrigger-684-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585927 DEBUG (simCloudManagerPool-683-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 585929 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 585934 DEBUG (simCloudManagerPool-683-thread-5) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 585934 DEBUG (simCloudManagerPool-683-thread-5) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 585946 DEBUG (ScheduledTrigger-684-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585946 DEBUG (simCloudManagerPool-683-thread-5) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 585977 DEBUG (ScheduledTrigger-684-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 585979 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 585979 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 3
   [junit4]   2> 585979 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 585979 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 585979 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 586022 DEBUG (ScheduledTrigger-684-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586042 DEBUG (ScheduledTrigger-684-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586062 DEBUG (ScheduledTrigger-684-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586074 DEBUG (ScheduledTrigger-684-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"ce7aef710cdbTd6ibpcr7nj7i3wzb94caqauea",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":227027398495451,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10000_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10000_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}}}}
   [junit4]   2> 586079 DEBUG (AutoscalingActionExecutor-685-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"ce7aef710cdbTd6ibpcr7nj7i3wzb94caqauea",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":227027398495451,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10000_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10000_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}},
   [junit4]   2>     "_enqueue_time_":227031431915651}}
   [junit4]   2> 586213 DEBUG (AutoscalingActionExecutor-685-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 6699 ms for event id=ce7aef710cdbTd6ibpcr7nj7i3wzb94caqauea
   [junit4]   2> 586216 DEBUG (ScheduledTrigger-684-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586334 DEBUG (ScheduledTrigger-684-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586336 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSearchRate
   [junit4]   2> 586336 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase
   [junit4]   2>
   [junit4]   2> 586336 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 586336 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
   [junit4]   2> 586336 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2>
   [junit4]   2> 586336 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 2
   [junit4]   2> 586336 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 0
   [junit4]   2> 586336 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
   [junit4]   2> 586337 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: [collection1]
   [junit4]   2> 586337 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 1
   [junit4]   2> 586337 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 1
   [junit4]   2> 586337 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 2
   [junit4]   2> 586338 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## * collection1 2
   [junit4]   2> 586342 DEBUG (ScheduledTrigger-684-thread-2) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"ce7bd6e7d0f9Td6ibpcr7nj7i3wzb94caqaued",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":227031281815801,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10000_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10000_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}}}}
   [junit4]   2> 586344 DEBUG (AutoscalingActionExecutor-685-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"ce7bd6e7d0f9Td6ibpcr7nj7i3wzb94caqaued",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":227031281815801,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10000_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10000_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}},
   [junit4]   2>     "_enqueue_time_":227044822262301}}
   [junit4]   2> 586345 DEBUG (AutoscalingActionExecutor-685-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 85 ms for event id=ce7bd6e7d0f9Td6ibpcr7nj7i3wzb94caqaued
   [junit4]   2> 586370 DEBUG (ScheduledTrigger-684-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586371 DEBUG (ScheduledTrigger-684-thread-3) [    ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
   [junit4]   2>   "id":"ce7dbcae663dTd6ibpcr7nj7i3wzb94caqaueg",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":227039431779901,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10000_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10000_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}}}}
   [junit4]   2> 586407 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - active           2
   [junit4]   2> 586407 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
   [junit4]   2> 586407 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - CREATE             1
   [junit4]   2> 586407 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling        2
   [junit4]   2> 586408 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update             4
   [junit4]   2> 586408 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
   [junit4]   2> 586408 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: search_rate_trigger
   [junit4]   2> 586409 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED         2
   [junit4]   2> 586409 INFO  (TEST-TestTriggerIntegration.testSearchRate-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED       2
   [junit4]   2> 586488 DEBUG (ScheduledTrigger-684-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586508 DEBUG (ScheduledTrigger-684-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586545 DEBUG (ScheduledTrigger-684-thread-4) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"ce7dbcae663dTd6ibpcr7nj7i3wzb94caqauei",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":227039431779901,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10000_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10000_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}}}}
   [junit4]   2> 586585 DEBUG (AutoscalingActionExecutor-685-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"ce7dbcae663dTd6ibpcr7nj7i3wzb94caqauei",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":227039431779901,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10000_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10000_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}},
   [junit4]   2>     "_enqueue_time_":227054974089401}}
   [junit4]   2> 586615 DEBUG (AutoscalingActionExecutor-685-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 1503 ms for event id=ce7dbcae663dTd6ibpcr7nj7i3wzb94caqauei
   [junit4]   2> 586715 DEBUG (ScheduledTrigger-684-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586715 DEBUG (ScheduledTrigger-684-thread-1) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"ce80ed097d29Td6ibpcr7nj7i3wzb94caqauel",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":227053127957801,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10000_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10000_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}}}}
   [junit4]   2> 586732 DEBUG (AutoscalingActionExecutor-685-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"ce80ed097d29Td6ibpcr7nj7i3wzb94caqauel",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":227053127957801,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10000_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    \"core\":\"collection1_shard1_replica_n1\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    \"core\":\"collection1_shard1_replica_n2\",\n    \"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    \"node_name\":\"127.0.0.1:10000_solr\",\n    \"state\":\"active\",\n    \"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}},
   [junit4]   2>     "_enqueue_time_":227063487668601}}
   [junit4]   2> 586734 DEBUG (AutoscalingActionExecutor-685-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 124 ms for event id=ce80ed097d29Td6ibpcr7nj7i3wzb94caqauel
   [junit4]   2> 586735 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
   [junit4]   2> 586747 DEBUG (ScheduledTrigger-684-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586747 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 586747 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 586748 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 586777 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 586777 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 586777 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
   [junit4]   2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 586804 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 586825 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 586825 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 586825 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 586825 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion -1
   [junit4]   2> 586825 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 586840 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 586840 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 586841 DEBUG (ScheduledTrigger-687-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586877 DEBUG (ScheduledTrigger-687-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586946 DEBUG (ScheduledTrigger-687-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 586950 DEBUG (simCloudManagerPool-686-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 586950 DEBUG (simCloudManagerPool-686-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 586950 DEBUG (simCloudManagerPool-686-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 586950 DEBUG (simCloudManagerPool-686-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 586950 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 586951 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 586951 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 586951 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 586951 DEBUG (ScheduledTrigger-687-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
   [junit4]   2> 586974 DEBUG (ScheduledTrigger-687-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 586974 DEBUG (ScheduledTrigger-687-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10001_solr
   [junit4]   2> 586974 DEBUG (ScheduledTrigger-687-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 586975 DEBUG (ScheduledTrigger-687-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10001_solr
   [junit4]   2> 587011 DEBUG (ScheduledTrigger-687-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 587027 DEBUG (ScheduledTrigger-687-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 587031 DEBUG (ScheduledTrigger-687-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 587048 DEBUG (ScheduledTrigger-687-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 587048 DEBUG (ScheduledTrigger-687-thread-3) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10001_solr]
   [junit4]   2> 587048 DEBUG (ScheduledTrigger-687-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"ce865a52b9b3Td6ibpcr7nj7i3wzb94caqaueo",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":227076436310451,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[227076436310451],
   [junit4]   2>     "nodeNames":["127.0.0.1:10001_solr"]}}
   [junit4]   2> 587048 DEBUG (AutoscalingActionExecutor-688-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"ce865a52b9b3Td6ibpcr7nj7i3wzb94caqaueo",
   [junit4]   2>   "source":"node_lost_trigger",
   [junit4]   2>   "eventTime":227076436310451,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[227076436310451],
   [junit4]   2>     "_enqueue_time_":227080101950901,
   [junit4]   2>     "nodeNames":["127.0.0.1:10001_solr"]}}
   [junit4]   2> 587050 DEBUG (simCloudManagerPool-686-thread-3) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 587050 DEBUG (simCloudManagerPool-686-thread-3) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 587050 DEBUG (simCloudManagerPool-686-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr]
   [junit4]   2> 587050 DEBUG (simCloudManagerPool-686-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr]
   [junit4]   2> 587050 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 587050 DEBUG (AutoscalingActionExecutor-688-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 72 ms for event id=ce865a52b9b3Td6ibpcr7nj7i3wzb94caqaueo
   [junit4]   2> 587050 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 587050 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 587051 DEBUG (ScheduledTrigger-687-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 587069 DEBUG (ScheduledTrigger-687-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 587071 DEBUG (ScheduledTrigger-687-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 587089 DEBUG (ScheduledTrigger-687-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase
   [junit4]   2>
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2>
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 1
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 1
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 1
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10001_solr
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling        2
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update             2
   [junit4]   2> 587090 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
   [junit4]   2> 587091 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger
   [junit4]   2> 587091 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED         1
   [junit4]   2> 587091 INFO  (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED       1
   [junit4]   2> 587096 DEBUG (ScheduledTrigger-687-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 587112 DEBUG (ScheduledTrigger-687-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
   [junit4]   2> 587112 INFO  (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[CFE48D7EA6F46476]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
   [junit4]   2> 587112 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 587112 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 587113 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 587113 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 587113 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 587115 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10001_solr
   [junit4]   2> 587115 INFO  (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 587115 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 587115 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[CFE48D7EA6F46476]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 587115 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 587116 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 587116 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 587116 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion -1
   [junit4]   2> 587116 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 587136 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 587136 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 587136 DEBUG (ScheduledTrigger-690-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 587156 DEBUG (ScheduledTrigger-690-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 587201 DEBUG (ScheduledTrigger-690-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 587216 DEBUG (simCloudManagerPool-689-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 587216 DEBUG (simCloudManagerPool-689-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
   [junit4]   2> 587274 DEBUG (ScheduledTrigger-690-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 587274 DEBUG (simCloudManagerPool-689-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 587274 DEBUG (simCloudManagerPool-689-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 587274 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 587274 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ]

[...truncated too long message...]

ucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_CFE48D7EA6F46476-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_CFE48D7EA6F46476-001/tempDir-002/hdfsBaseDir/data/data4/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:35608) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1042007198-127.0.0.1-1515778082972 (Datanode Uuid ca620728-9495-4737-a889-3cd1eef7329e) service to lucene2-us-west.apache.org/127.0.0.1:35608
   [junit4]   2> 6040637 INFO  (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@76aacac) [    ] BlockStateChange BLOCK* BlockManager: ask 127.0.0.1:45523 to delete [blk_1073741856_1032, blk_1073741857_1033, blk_1073741826_1002, blk_1073741830_1006, blk_1073741831_1007, blk_1073741832_1008, blk_1073741833_1009, blk_1073741834_1010, blk_1073741835_1011, blk_1073741836_1012, blk_1073741837_1013, blk_1073741838_1014, blk_1073741839_1015, blk_1073741840_1016, blk_1073741841_1017, blk_1073741842_1018, blk_1073741843_1019, blk_1073741844_1020, blk_1073741845_1021, blk_1073741846_1022, blk_1073741847_1023, blk_1073741848_1024]
   [junit4]   2> 6040666 WARN  (SUITE-HdfsRecoveryZkTest-seed#[CFE48D7EA6F46476]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 6040802 INFO  (SUITE-HdfsRecoveryZkTest-seed#[CFE48D7EA6F46476]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 6040939 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_CFE48D7EA6F46476-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_CFE48D7EA6F46476-001/tempDir-002/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:35608) [    ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
   [junit4]   2> 6040939 WARN  (DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_CFE48D7EA6F46476-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_CFE48D7EA6F46476-001/tempDir-002/hdfsBaseDir/data/data2/]]  heartbeating to lucene2-us-west.apache.org/127.0.0.1:35608) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1042007198-127.0.0.1-1515778082972 (Datanode Uuid bae5d105-4775-4366-a20d-92e6a594dc14) service to lucene2-us-west.apache.org/127.0.0.1:35608
   [junit4]   2> 6040964 INFO  (SUITE-HdfsRecoveryZkTest-seed#[CFE48D7EA6F46476]-worker) [    ] o.m.log Stopped HttpServer2$[hidden email]:0
   [junit4]   2> 6041151 INFO  (SUITE-HdfsRecoveryZkTest-seed#[CFE48D7EA6F46476]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37516 37516
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_CFE48D7EA6F46476-001
   [junit4]   2> Jan 12, 2018 5:28:42 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {rnd_b=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_t=PostingsFormat(name=MockRandom), a_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{_version_=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=164, maxMBSortInHeap=7.334855975779313, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@4368135b), locale=hu, timezone=America/Asuncion
   [junit4]   2> NOTE: Linux 4.4.0-104-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=2,free=152170760,total=485490688
   [junit4]   2> NOTE: All tests run in this JVM: [DeleteNodeTest, SolrTestCaseJ4Test, TestNestedDocsSort, TestConfigSetsAPIExclusivity, BlobRepositoryCloudTest, ZkNodePropsTest, DirectSolrConnectionTest, TestLFUCache, TestSubQueryTransformer, TestGeoJSONResponseWriter, TestCoreContainer, ExitableDirectoryReaderTest, UpdateLogTest, TestReloadAndDeleteDocs, DistanceFunctionTest, TestSchemalessBufferedUpdates, TestLegacyFieldCache, StatsComponentTest, TestSSLRandomization, OverseerTest, NodeAddedTriggerTest, UnloadDistributedZkTest, HdfsBasicDistributedZk2Test, CdcrUpdateLogTest, TestTrieFacet, V2ApiIntegrationTest, TestNamedUpdateProcessors, SolrGraphiteReporterTest, TestNodeAddedTrigger, MetricUtilsTest, TestCloudJSONFacetJoinDomain, ClusterStateTest, CoreAdminHandlerTest, CustomCollectionTest, CopyFieldTest, TestSmileRequest, TestExclusionRuleCollectionAccess, ConfigSetsAPITest, TestRetrieveFieldsOptimizer, TestSerializedLuceneMatchVersion, TestPrepRecovery, InfoHandlerTest, TestMultiValuedNumericRangeQuery, LoggingHandlerTest, RecoveryZkTest, TestSolrCLIRunExample, HdfsSyncSliceTest, SuggestComponentContextFilterQueryTest, TestSchemaResource, TestFileDictionaryLookup, OverseerRolesTest, TestFieldCacheSortRandom, WordBreakSolrSpellCheckerTest, BasicDistributedZk2Test, UUIDUpdateProcessorFallbackTest, DistributedExpandComponentTest, TestReloadDeadlock, TestRestoreCore, TestSimDistributedQueue, SpellPossibilityIteratorTest, TestLMDirichletSimilarityFactory, TestLockTree, ConnectionReuseTest, ReplaceNodeTest, TestLeaderElectionZkExpiry, SortByFunctionTest, HttpPartitionTest, TestSizeLimitedDistributedMap, ShardSplitTest, ScriptEngineTest, DistribJoinFromCollectionTest, TestSolrConfigHandler, DocExpirationUpdateProcessorFactoryTest, TlogReplayBufferedWhileIndexingTest, SystemLogListenerTest, ManagedSchemaRoundRobinCloudTest, TestTolerantUpdateProcessorCloud, TestIndexSearcher, OverriddenZkACLAndCredentialsProvidersTest, HdfsUnloadDistributedZkTest, TestSolrCoreProperties, TestGroupingSearch, RegexBytesRefFilterTest, AnalyticsMergeStrategyTest, TestDistributedGrouping, TestDynamicFieldResource, AtomicUpdateProcessorFactoryTest, SolrCLIZkUtilsTest, TestSolrCloudWithKerberosAlt, SecurityConfHandlerTest, SortSpecParsingTest, UUIDFieldTest, ClassificationUpdateProcessorIntegrationTest, TestRandomFlRTGCloud, BitVectorTest, PluginInfoTest, XsltUpdateRequestHandlerTest, TestElisionMultitermQuery, TestBackupRepositoryFactory, TestQuerySenderListener, TestMinMaxOnMultiValuedField, TestClusterStateProvider, TestEmbeddedSolrServerConstructors, SolrCoreCheckLockOnStartupTest, HdfsRecoverLeaseTest, MetricsConfigTest, CdcrReplicationHandlerTest, CdcrBidirectionalTest, QueryElevationComponentTest, TestImpersonationWithHadoopAuth, ConfigureRecoveryStrategyTest, PreAnalyzedUpdateProcessorTest, TestLRUCache, EchoParamsTest, TestSolrFieldCacheBean, TestCustomStream, TestMacros, TestPerFieldSimilarityWithDefaultOverride, TestUninvertingReader, TestBinaryResponseWriter, SolrSlf4jReporterTest, TestApiFramework, TestDeleteCollectionOnDownNodes, SyncSliceTest, FullSolrCloudDistribCmdsTest, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, TestRandomDVFaceting, TestRealTimeGet, TestStressReorder, TestReload, SimpleFacetsTest, SolrCoreTest, SpellCheckComponentTest, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTest, TestFiltering, TestFunctionQuery, TestBadConfig, HighlighterTest, SimplePostToolTest, TestCoreDiscovery, TestExtendedDismaxParser, SolrRequestParserTest, DocValuesTest, DocumentAnalysisRequestHandlerTest, TermsComponentTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, BadComponentTest, TestSolrDeletionPolicy2, TestDocSet, TestBinaryField, TestSearchPerf, NumericFieldsTest, OutputWriterTest, TestCharFilters, TestXIncludeConfig, TestBM25SimilarityFactory, TestFastOutputStream, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, DateFieldTest, DOMUtilTest, TestUtils, SystemInfoHandlerTest, TestRTGBase, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestTolerantSearch, TestEmbeddedSolrServerAdminHandler, TestJettySolrRunner, AliasIntegrationTest, AsyncCallRequestStatusResponseTest, CollectionReloadTest, CollectionTooManyReplicasTest, DeleteLastCustomShardedReplicaTest, DeleteStatusTest, LeaderFailureAfterFreshStartTest, PeerSyncReplicationTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudRecovery, TestConfigSetsAPIZkFailure, TestDistribDocBasedVersion, TestDownShardTolerantSearch, TestHdfsCloudBackupRestore, TestOnReconnectListenerSupport, TestRandomRequestDistribution, TestReplicaProperties, TestShortCircuitedRequests, TestStressCloudBlindAtomicUpdates, TestUtilizeNode, AutoScalingHandlerTest, HdfsAutoAddReplicasIntegrationTest, CdcrRequestHandlerTest, HdfsNNFailoverTest, HdfsRecoveryZkTest]
   [junit4] Completed [583/767 (5!)] on J2 in 49.57s, 1 test, 1 failure <<< FAILURES!

[...truncated 9258 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:844: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:788: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid25196.hprof

Total time: 475 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1448 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1448/

2 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.sim.TestClusterStateProvider.testAutoScalingConfig

Error Message:
expected: org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig<{   "cluster-preferences":[{"maximize":"freedisk"}],   "triggers":{".auto_add_replicas":{       "name":".auto_add_replicas",       "event":"nodeLost",       "waitFor":30,       "actions":[         {           "name":"auto_add_replicas_plan",           "class":"solr.AutoAddReplicasPlanAction"},         {           "name":"execute_plan",           "class":"solr.ExecutePlanAction"}],       "enabled":true}},   "listeners":{".auto_add_replicas.system":{       "trigger":".auto_add_replicas",       "afterAction":[],       "stage":[         "STARTED",         "ABORTED",         "SUCCEEDED",         "FAILED",         "BEFORE_ACTION",         "AFTER_ACTION",         "IGNORED"],       "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",       "beforeAction":[]}},   "properties":{}}> but was: org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig<{   "cluster-preferences":[{"maximize":"freedisk"}],   "triggers":{".auto_add_replicas":{       "name":".auto_add_replicas",       "event":"nodeLost",       "waitFor":30,       "actions":[         {           "name":"auto_add_replicas_plan",           "class":"solr.AutoAddReplicasPlanAction"},         {           "name":"execute_plan",           "class":"solr.ExecutePlanAction"}],       "enabled":true}},   "listeners":{".auto_add_replicas.system":{       "trigger":".auto_add_replicas",       "afterAction":[],       "stage":[         "STARTED",         "ABORTED",         "SUCCEEDED",         "FAILED",         "BEFORE_ACTION",         "AFTER_ACTION",         "IGNORED"],       "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",       "beforeAction":[]}},   "properties":{}}>

Stack Trace:
java.lang.AssertionError: expected: org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig<{
  "cluster-preferences":[{"maximize":"freedisk"}],
  "triggers":{".auto_add_replicas":{
      "name":".auto_add_replicas",
      "event":"nodeLost",
      "waitFor":30,
      "actions":[
        {
          "name":"auto_add_replicas_plan",
          "class":"solr.AutoAddReplicasPlanAction"},
        {
          "name":"execute_plan",
          "class":"solr.ExecutePlanAction"}],
      "enabled":true}},
  "listeners":{".auto_add_replicas.system":{
      "trigger":".auto_add_replicas",
      "afterAction":[],
      "stage":[
        "STARTED",
        "ABORTED",
        "SUCCEEDED",
        "FAILED",
        "BEFORE_ACTION",
        "AFTER_ACTION",
        "IGNORED"],
      "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",
      "beforeAction":[]}},
  "properties":{}}> but was: org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig<{
  "cluster-preferences":[{"maximize":"freedisk"}],
  "triggers":{".auto_add_replicas":{
      "name":".auto_add_replicas",
      "event":"nodeLost",
      "waitFor":30,
      "actions":[
        {
          "name":"auto_add_replicas_plan",
          "class":"solr.AutoAddReplicasPlanAction"},
        {
          "name":"execute_plan",
          "class":"solr.ExecutePlanAction"}],
      "enabled":true}},
  "listeners":{".auto_add_replicas.system":{
      "trigger":".auto_add_replicas",
      "afterAction":[],
      "stage":[
        "STARTED",
        "ABORTED",
        "SUCCEEDED",
        "FAILED",
        "BEFORE_ACTION",
        "AFTER_ACTION",
        "IGNORED"],
      "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",
      "beforeAction":[]}},
  "properties":{}}>
        at __randomizedtesting.SeedInfo.seed([456CF5C9CB746056:7AE4F661DC2290B1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.cloud.autoscaling.sim.TestClusterStateProvider.testAutoScalingConfig(TestClusterStateProvider.java:214)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1515903644868,generation=2,filelist=[_64.fdt, _64.fdx, _64.fnm, _64.nvd, _64.nvm, _64.si, _64_FST50_0.doc, _64_FST50_0.tfp, _65.fdt, _65.fdx, _65.fnm, _65.nvd, _65.nvm, _65.si, _65_FST50_0.doc, _65_FST50_0.tfp, _66.fdt, _66.fdx, _66.fnm, _66.nvd, _66.nvm, _66.si, _66_FST50_0.doc, _66_FST50_0.tfp, _67.fdt, _67.fdx, _67.fnm, _67.nvd, _67.nvm, _67.si, _67_FST50_0.doc, _67_FST50_0.tfp, _68.fdt, _68.fdx, _68.fnm, _68.nvd, _68.nvm, _68.si, _68_FST50_0.doc, _68_FST50_0.tfp, _69.fdt, _69.fdx, _69.fnm, _69.nvd, _69.nvm, _69.si, _69_FST50_0.doc, _69_FST50_0.tfp, _6a.fdt, _6a.fdx, _6a.fnm, _6a.nvd, _6a.nvm, _6a.si, _6a_FST50_0.doc, _6a_FST50_0.tfp, _6b.fdt, _6b.fdx, _6b.fnm, _6b.nvd, _6b.nvm, _6b.si, _6b_FST50_0.doc, _6b_FST50_0.tfp, _6c.fdt, _6c.fdx, _6c.fnm, _6c.nvd, _6c.nvm, _6c.si, _6c_FST50_0.doc, _6c_FST50_0.tfp, _6d.fdt, _6d.fdx, _6d.fnm, _6d.nvd, _6d.nvm, _6d.si, _6d_FST50_0.doc, _6d_FST50_0.tfp, _6e.fdt, _6e.fdx, _6e.fnm, _6e.nvd, _6e.nvm, _6e.si, _6e_FST50_0.doc, _6e_FST50_0.tfp, _6g.fdt, _6g.fdx, _6g.fnm, _6g.nvd, _6g.nvm, _6g.si, _6g_FST50_0.doc, _6g_FST50_0.tfp, _6h.fdt, _6h.fdx, _6h.fnm, _6h.nvd, _6h.nvm, _6h.si, _6h_FST50_0.doc, _6h_FST50_0.tfp, _6i.fdt, _6i.fdx, _6i.fnm, _6i.nvd, _6i.nvm, _6i.si, _6i_FST50_0.doc, _6i_FST50_0.tfp, _6j.fdt, _6j.fdx, _6j.fnm, _6j.nvd, _6j.nvm, _6j.si, _6j_FST50_0.doc, _6j_FST50_0.tfp, segments_2]}]> but was:<[{indexVersion=1515903644868,generation=2,filelist=[_64.fdt, _64.fdx, _64.fnm, _64.nvd, _64.nvm, _64.si, _64_FST50_0.doc, _64_FST50_0.tfp, _65.fdt, _65.fdx, _65.fnm, _65.nvd, _65.nvm, _65.si, _65_FST50_0.doc, _65_FST50_0.tfp, _66.fdt, _66.fdx, _66.fnm, _66.nvd, _66.nvm, _66.si, _66_FST50_0.doc, _66_FST50_0.tfp, _67.fdt, _67.fdx, _67.fnm, _67.nvd, _67.nvm, _67.si, _67_FST50_0.doc, _67_FST50_0.tfp, _68.fdt, _68.fdx, _68.fnm, _68.nvd, _68.nvm, _68.si, _68_FST50_0.doc, _68_FST50_0.tfp, _69.fdt, _69.fdx, _69.fnm, _69.nvd, _69.nvm, _69.si, _69_FST50_0.doc, _69_FST50_0.tfp, _6a.fdt, _6a.fdx, _6a.fnm, _6a.nvd, _6a.nvm, _6a.si, _6a_FST50_0.doc, _6a_FST50_0.tfp, _6b.fdt, _6b.fdx, _6b.fnm, _6b.nvd, _6b.nvm, _6b.si, _6b_FST50_0.doc, _6b_FST50_0.tfp, _6c.fdt, _6c.fdx, _6c.fnm, _6c.nvd, _6c.nvm, _6c.si, _6c_FST50_0.doc, _6c_FST50_0.tfp, _6d.fdt, _6d.fdx, _6d.fnm, _6d.nvd, _6d.nvm, _6d.si, _6d_FST50_0.doc, _6d_FST50_0.tfp, _6e.fdt, _6e.fdx, _6e.fnm, _6e.nvd, _6e.nvm, _6e.si, _6e_FST50_0.doc, _6e_FST50_0.tfp, _6g.fdt, _6g.fdx, _6g.fnm, _6g.nvd, _6g.nvm, _6g.si, _6g_FST50_0.doc, _6g_FST50_0.tfp, _6h.fdt, _6h.fdx, _6h.fnm, _6h.nvd, _6h.nvm, _6h.si, _6h_FST50_0.doc, _6h_FST50_0.tfp, _6i.fdt, _6i.fdx, _6i.fnm, _6i.nvd, _6i.nvm, _6i.si, _6i_FST50_0.doc, _6i_FST50_0.tfp, _6j.fdt, _6j.fdx, _6j.fnm, _6j.nvd, _6j.nvm, _6j.si, _6j_FST50_0.doc, _6j_FST50_0.tfp, segments_2]}, {indexVersion=1515903644868,generation=3,filelist=[_6e.fdt, _6e.fdx, _6e.fnm, _6e.nvd, _6e.nvm, _6e.si, _6e_FST50_0.doc, _6e_FST50_0.tfp, _6f.fdt, _6f.fdx, _6f.fnm, _6f.nvd, _6f.nvm, _6f.si, _6f_FST50_0.doc, _6f_FST50_0.tfp, _6g.fdt, _6g.fdx, _6g.fnm, _6g.nvd, _6g.nvm, _6g.si, _6g_FST50_0.doc, _6g_FST50_0.tfp, _6h.fdt, _6h.fdx, _6h.fnm, _6h.nvd, _6h.nvm, _6h.si, _6h_FST50_0.doc, _6h_FST50_0.tfp, _6i.fdt, _6i.fdx, _6i.fnm, _6i.nvd, _6i.nvm, _6i.si, _6i_FST50_0.doc, _6i_FST50_0.tfp, _6j.fdt, _6j.fdx, _6j.fnm, _6j.nvd, _6j.nvm, _6j.si, _6j_FST50_0.doc, _6j_FST50_0.tfp, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1515903644868,generation=2,filelist=[_64.fdt, _64.fdx, _64.fnm, _64.nvd, _64.nvm, _64.si, _64_FST50_0.doc, _64_FST50_0.tfp, _65.fdt, _65.fdx, _65.fnm, _65.nvd, _65.nvm, _65.si, _65_FST50_0.doc, _65_FST50_0.tfp, _66.fdt, _66.fdx, _66.fnm, _66.nvd, _66.nvm, _66.si, _66_FST50_0.doc, _66_FST50_0.tfp, _67.fdt, _67.fdx, _67.fnm, _67.nvd, _67.nvm, _67.si, _67_FST50_0.doc, _67_FST50_0.tfp, _68.fdt, _68.fdx, _68.fnm, _68.nvd, _68.nvm, _68.si, _68_FST50_0.doc, _68_FST50_0.tfp, _69.fdt, _69.fdx, _69.fnm, _69.nvd, _69.nvm, _69.si, _69_FST50_0.doc, _69_FST50_0.tfp, _6a.fdt, _6a.fdx, _6a.fnm, _6a.nvd, _6a.nvm, _6a.si, _6a_FST50_0.doc, _6a_FST50_0.tfp, _6b.fdt, _6b.fdx, _6b.fnm, _6b.nvd, _6b.nvm, _6b.si, _6b_FST50_0.doc, _6b_FST50_0.tfp, _6c.fdt, _6c.fdx, _6c.fnm, _6c.nvd, _6c.nvm, _6c.si, _6c_FST50_0.doc, _6c_FST50_0.tfp, _6d.fdt, _6d.fdx, _6d.fnm, _6d.nvd, _6d.nvm, _6d.si, _6d_FST50_0.doc, _6d_FST50_0.tfp, _6e.fdt, _6e.fdx, _6e.fnm, _6e.nvd, _6e.nvm, _6e.si, _6e_FST50_0.doc, _6e_FST50_0.tfp, _6g.fdt, _6g.fdx, _6g.fnm, _6g.nvd, _6g.nvm, _6g.si, _6g_FST50_0.doc, _6g_FST50_0.tfp, _6h.fdt, _6h.fdx, _6h.fnm, _6h.nvd, _6h.nvm, _6h.si, _6h_FST50_0.doc, _6h_FST50_0.tfp, _6i.fdt, _6i.fdx, _6i.fnm, _6i.nvd, _6i.nvm, _6i.si, _6i_FST50_0.doc, _6i_FST50_0.tfp, _6j.fdt, _6j.fdx, _6j.fnm, _6j.nvd, _6j.nvm, _6j.si, _6j_FST50_0.doc, _6j_FST50_0.tfp, segments_2]}]> but was:<[{indexVersion=1515903644868,generation=2,filelist=[_64.fdt, _64.fdx, _64.fnm, _64.nvd, _64.nvm, _64.si, _64_FST50_0.doc, _64_FST50_0.tfp, _65.fdt, _65.fdx, _65.fnm, _65.nvd, _65.nvm, _65.si, _65_FST50_0.doc, _65_FST50_0.tfp, _66.fdt, _66.fdx, _66.fnm, _66.nvd, _66.nvm, _66.si, _66_FST50_0.doc, _66_FST50_0.tfp, _67.fdt, _67.fdx, _67.fnm, _67.nvd, _67.nvm, _67.si, _67_FST50_0.doc, _67_FST50_0.tfp, _68.fdt, _68.fdx, _68.fnm, _68.nvd, _68.nvm, _68.si, _68_FST50_0.doc, _68_FST50_0.tfp, _69.fdt, _69.fdx, _69.fnm, _69.nvd, _69.nvm, _69.si, _69_FST50_0.doc, _69_FST50_0.tfp, _6a.fdt, _6a.fdx, _6a.fnm, _6a.nvd, _6a.nvm, _6a.si, _6a_FST50_0.doc, _6a_FST50_0.tfp, _6b.fdt, _6b.fdx, _6b.fnm, _6b.nvd, _6b.nvm, _6b.si, _6b_FST50_0.doc, _6b_FST50_0.tfp, _6c.fdt, _6c.fdx, _6c.fnm, _6c.nvd, _6c.nvm, _6c.si, _6c_FST50_0.doc, _6c_FST50_0.tfp, _6d.fdt, _6d.fdx, _6d.fnm, _6d.nvd, _6d.nvm, _6d.si, _6d_FST50_0.doc, _6d_FST50_0.tfp, _6e.fdt, _6e.fdx, _6e.fnm, _6e.nvd, _6e.nvm, _6e.si, _6e_FST50_0.doc, _6e_FST50_0.tfp, _6g.fdt, _6g.fdx, _6g.fnm, _6g.nvd, _6g.nvm, _6g.si, _6g_FST50_0.doc, _6g_FST50_0.tfp, _6h.fdt, _6h.fdx, _6h.fnm, _6h.nvd, _6h.nvm, _6h.si, _6h_FST50_0.doc, _6h_FST50_0.tfp, _6i.fdt, _6i.fdx, _6i.fnm, _6i.nvd, _6i.nvm, _6i.si, _6i_FST50_0.doc, _6i_FST50_0.tfp, _6j.fdt, _6j.fdx, _6j.fnm, _6j.nvd, _6j.nvm, _6j.si, _6j_FST50_0.doc, _6j_FST50_0.tfp, segments_2]}, {indexVersion=1515903644868,generation=3,filelist=[_6e.fdt, _6e.fdx, _6e.fnm, _6e.nvd, _6e.nvm, _6e.si, _6e_FST50_0.doc, _6e_FST50_0.tfp, _6f.fdt, _6f.fdx, _6f.fnm, _6f.nvd, _6f.nvm, _6f.si, _6f_FST50_0.doc, _6f_FST50_0.tfp, _6g.fdt, _6g.fdx, _6g.fnm, _6g.nvd, _6g.nvm, _6g.si, _6g_FST50_0.doc, _6g_FST50_0.tfp, _6h.fdt, _6h.fdx, _6h.fnm, _6h.nvd, _6h.nvm, _6h.si, _6h_FST50_0.doc, _6h_FST50_0.tfp, _6i.fdt, _6i.fdx, _6i.fnm, _6i.nvd, _6i.nvm, _6i.si, _6i_FST50_0.doc, _6i_FST50_0.tfp, _6j.fdt, _6j.fdx, _6j.fnm, _6j.nvd, _6j.nvm, _6j.si, _6j_FST50_0.doc, _6j_FST50_0.tfp, segments_3]}]>
        at __randomizedtesting.SeedInfo.seed([456CF5C9CB746056:60BBEEF9BB3C6E55]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1277)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13043 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestClusterStateProvider
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/init-core-data-001
   [junit4]   2> 1770090 WARN  (SUITE-TestClusterStateProvider-seed#[456CF5C9CB746056]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 1770135 INFO  (SUITE-TestClusterStateProvider-seed#[456CF5C9CB746056]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1770136 INFO  (SUITE-TestClusterStateProvider-seed#[456CF5C9CB746056]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1770136 INFO  (SUITE-TestClusterStateProvider-seed#[456CF5C9CB746056]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1770137 INFO  (SUITE-TestClusterStateProvider-seed#[456CF5C9CB746056]-worker) [    ] o.a.s.c.a.s.TestClusterStateProvider ####### Using simulated components? true
   [junit4]   2> 1770137 INFO  (SUITE-TestClusterStateProvider-seed#[456CF5C9CB746056]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001
   [junit4]   2> 1770137 INFO  (SUITE-TestClusterStateProvider-seed#[456CF5C9CB746056]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1770137 INFO  (Thread-852) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1770137 INFO  (Thread-852) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1770184 ERROR (Thread-852) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1770237 INFO  (SUITE-TestClusterStateProvider-seed#[456CF5C9CB746056]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:35600
   [junit4]   2> 1770239 INFO  (zkConnectionManagerCallback-1727-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1770242 INFO  (jetty-launcher-1724-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1770242 INFO  (jetty-launcher-1724-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1770242 INFO  (jetty-launcher-1724-thread-3) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1770243 INFO  (jetty-launcher-1724-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@32f20dc9{/solr,null,AVAILABLE}
   [junit4]   2> 1770243 INFO  (jetty-launcher-1724-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5e9ff51a{/solr,null,AVAILABLE}
   [junit4]   2> 1770244 INFO  (jetty-launcher-1724-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1b286bf2{/solr,null,AVAILABLE}
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@510511b2{HTTP/1.1,[http/1.1]}{127.0.0.1:44301}
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@240ca53d{HTTP/1.1,[http/1.1]}{127.0.0.1:39455}
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-1) [    ] o.e.j.s.Server Started @1785675ms
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-3) [    ] o.e.j.s.Server Started @1785675ms
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44301}
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39455}
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@161340b1{HTTP/1.1,[http/1.1]}{127.0.0.1:41593}
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-2) [    ] o.e.j.s.Server Started @1785675ms
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41593}
   [junit4]   2> 1770247 ERROR (jetty-launcher-1724-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1770247 ERROR (jetty-launcher-1724-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1770247 ERROR (jetty-launcher-1724-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-14T03:36:18.501Z
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-14T03:36:18.501Z
   [junit4]   2> 1770247 INFO  (jetty-launcher-1724-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-14T03:36:18.501Z
   [junit4]   2> 1770265 INFO  (zkConnectionManagerCallback-1730-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1770265 INFO  (zkConnectionManagerCallback-1731-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1770265 INFO  (zkConnectionManagerCallback-1733-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1770265 INFO  (jetty-launcher-1724-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1770265 INFO  (jetty-launcher-1724-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1770265 INFO  (jetty-launcher-1724-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1770290 INFO  (jetty-launcher-1724-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35600/solr
   [junit4]   2> 1770290 INFO  (jetty-launcher-1724-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35600/solr
   [junit4]   2> 1770291 INFO  (jetty-launcher-1724-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35600/solr
   [junit4]   2> 1770291 INFO  (zkConnectionManagerCallback-1740-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1770292 INFO  (zkConnectionManagerCallback-1742-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1770294 INFO  (zkConnectionManagerCallback-1745-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1770295 INFO  (zkConnectionManagerCallback-1747-thread-1-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1770295 INFO  (zkConnectionManagerCallback-1749-thread-1-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1770295 INFO  (zkConnectionManagerCallback-1751-thread-1-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1770546 INFO  (jetty-launcher-1724-thread-2) [n:127.0.0.1:41593_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1770546 INFO  (jetty-launcher-1724-thread-3) [n:127.0.0.1:39455_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1770546 INFO  (jetty-launcher-1724-thread-2) [n:127.0.0.1:41593_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41593_solr
   [junit4]   2> 1770547 INFO  (jetty-launcher-1724-thread-3) [n:127.0.0.1:39455_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39455_solr
   [junit4]   2> 1770547 INFO  (jetty-launcher-1724-thread-2) [n:127.0.0.1:41593_solr    ] o.a.s.c.Overseer Overseer (id=99346086519898121-127.0.0.1:41593_solr-n_0000000000) starting
   [junit4]   2> 1770561 INFO  (zkCallback-1750-thread-1-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1770561 INFO  (zkCallback-1748-thread-1-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1770563 INFO  (jetty-launcher-1724-thread-1) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1770574 INFO  (jetty-launcher-1724-thread-1) [n:127.0.0.1:44301_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1770576 INFO  (jetty-launcher-1724-thread-2) [n:127.0.0.1:41593_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41593_solr
   [junit4]   2> 1770576 INFO  (jetty-launcher-1724-thread-1) [n:127.0.0.1:44301_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44301_solr
   [junit4]   2> 1770589 INFO  (zkCallback-1748-thread-1-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1770621 INFO  (zkCallback-1748-thread-1-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1770621 INFO  (zkCallback-1746-thread-1-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 1770621 INFO  (zkCallback-1750-thread-1-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 1770946 INFO  (jetty-launcher-1724-thread-1) [n:127.0.0.1:44301_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44301.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1770967 INFO  (jetty-launcher-1724-thread-3) [n:127.0.0.1:39455_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39455.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1770967 INFO  (jetty-launcher-1724-thread-2) [n:127.0.0.1:41593_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41593.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1770983 INFO  (jetty-launcher-1724-thread-1) [n:127.0.0.1:44301_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44301.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1770983 INFO  (jetty-launcher-1724-thread-1) [n:127.0.0.1:44301_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44301.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1770983 INFO  (jetty-launcher-1724-thread-3) [n:127.0.0.1:39455_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39455.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1770983 INFO  (jetty-launcher-1724-thread-3) [n:127.0.0.1:39455_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39455.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1770984 INFO  (jetty-launcher-1724-thread-1) [n:127.0.0.1:44301_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001/node1/.
   [junit4]   2> 1770984 INFO  (jetty-launcher-1724-thread-3) [n:127.0.0.1:39455_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001/node3/.
   [junit4]   2> 1770992 INFO  (jetty-launcher-1724-thread-2) [n:127.0.0.1:41593_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41593.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1770992 INFO  (jetty-launcher-1724-thread-2) [n:127.0.0.1:41593_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41593.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1770993 INFO  (jetty-launcher-1724-thread-2) [n:127.0.0.1:41593_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001/node2/.
   [junit4]   2> 1771122 INFO  (zkConnectionManagerCallback-1759-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1771124 INFO  (zkConnectionManagerCallback-1763-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1771136 INFO  (SUITE-TestClusterStateProvider-seed#[456CF5C9CB746056]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1771137 INFO  (SUITE-TestClusterStateProvider-seed#[456CF5C9CB746056]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35600/solr ready
   [junit4]   2> 1771141 INFO  (qtp141082257-5701) [n:127.0.0.1:41593_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=1&replicationFactor=2&name=.system&nrtReplicas=2&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1771153 INFO  (OverseerThreadFactory-2133-thread-1-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.CreateCollectionCmd Create collection .system
   [junit4]   2> 1771258 INFO  (OverseerStateUpdate-99346086519898121-127.0.0.1:41593_solr-n_0000000000) [n:127.0.0.1:41593_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":".system",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":".system_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:44301/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1771259 INFO  (OverseerStateUpdate-99346086519898121-127.0.0.1:41593_solr-n_0000000000) [n:127.0.0.1:41593_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":".system",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":".system_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:39455/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1771261 INFO  (OverseerStateUpdate-99346086519898121-127.0.0.1:41593_solr-n_0000000000) [n:127.0.0.1:41593_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":".system",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":".system_shard1_replica_p4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:41593/solr",
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"}
   [junit4]   2> 1771464 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n1&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1771464 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=.system&newCollection=true&name=.system_shard1_replica_p4&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 1771465 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1771466 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1771466 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n2&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1771466 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1771598 INFO  (zkCallback-1748-thread-1-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1771598 INFO  (zkCallback-1750-thread-1-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1771598 INFO  (zkCallback-1746-thread-1-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1771598 INFO  (zkCallback-1748-thread-2-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1771598 INFO  (zkCallback-1750-thread-2-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1771599 INFO  (zkCallback-1746-thread-2-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1772491 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1772495 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 1772497 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema [.system_shard1_replica_n1] Schema name=_system collection or core
   [junit4]   2> 1772524 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 1772525 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/.system/managed-schema
   [junit4]   2> 1772527 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1772528 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1772542 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/.system/schema.xml to /configs/.system/schema.xml.bak
   [junit4]   2> 1772543 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n1' using configuration from collection .system, trusted=true
   [junit4]   2> 1772579 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44301.solr.core..system.shard1.replica_n1' (registry 'solr.core..system.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1772580 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1772580 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SolrCore [[.system_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001/node1/.system_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001/node1/./.system_shard1_replica_n1/data/]
   [junit4]   2> 1772583 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.s.IndexSchema [.system_shard1_replica_p4] Schema name=_system collection or core
   [junit4]   2> 1772598 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 1772598 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_p4' using configuration from collection .system, trusted=true
   [junit4]   2> 1772599 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41593.solr.core..system.shard1.replica_p4' (registry 'solr.core..system.shard1.replica_p4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1772599 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1772599 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.c.SolrCore [[.system_shard1_replica_p4] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001/node2/.system_shard1_replica_p4], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001/node2/./.system_shard1_replica_p4/data/]
   [junit4]   2> 1772685 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.IndexSchema [.system_shard1_replica_n2] Schema name=_system collection or core
   [junit4]   2> 1772688 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 1772688 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n2' using configuration from collection .system, trusted=true
   [junit4]   2> 1772688 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39455.solr.core..system.shard1.replica_n2' (registry 'solr.core..system.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1772689 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1772689 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.SolrCore [[.system_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001/node3/.system_shard1_replica_n2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001/node3/./.system_shard1_replica_n2/data/]
   [junit4]   2> 1772766 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs
   [junit4]   2> 1772766 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1772766 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1772766 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1772767 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs
   [junit4]   2> 1772767 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1772767 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.s.SolrIndexSearcher Opening [Searcher@160fa285[.system_shard1_replica_p4] main]
   [junit4]   2> 1772768 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
   [junit4]   2> 1772768 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5c5076e3[.system_shard1_replica_n1] main]
   [junit4]   2> 1772768 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
   [junit4]   2> 1772776 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
   [junit4]   2> 1772777 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 1772777 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
   [junit4]   2> 1772777 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
   [junit4]   2> 1772777 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
   [junit4]   2> 1772777 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1772777 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 1772778 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.c.ZkController .system_shard1_replica_p4 starting background replication from leader
   [junit4]   2> 1772778 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1772778 INFO  (searcherExecutor-2141-thread-1-processing-n:127.0.0.1:41593_solr x:.system_shard1_replica_p4 s:shard1 c:.system r:core_node6) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.c.SolrCore [.system_shard1_replica_p4] Registered new searcher Searcher@160fa285[.system_shard1_replica_p4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1772779 INFO  (searcherExecutor-2140-thread-1-processing-n:127.0.0.1:44301_solr x:.system_shard1_replica_n1 s:shard1 c:.system r:core_node3) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SolrCore [.system_shard1_replica_n1] Registered new searcher Searcher@5c5076e3[.system_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1772779 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1589537387087659008
   [junit4]   2> 1772779 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
   [junit4]   2> 1772783 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1772797 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
   [junit4]   2> 1772797 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1772824 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1772824 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1772825 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs
   [junit4]   2> 1772825 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1772826 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@613f52da[.system_shard1_replica_n2] main]
   [junit4]   2> 1772827 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
   [junit4]   2> 1772827 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
   [junit4]   2> 1772827 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
   [junit4]   2> 1772827 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 1772828 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1772829 INFO  (searcherExecutor-2142-thread-1-processing-n:127.0.0.1:39455_solr x:.system_shard1_replica_n2 s:shard1 c:.system r:core_node5) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.SolrCore [.system_shard1_replica_n2] Registered new searcher Searcher@613f52da[.system_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1772829 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1589537387140087808
   [junit4]   2> 1772902 INFO  (zkCallback-1748-thread-2-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1772902 INFO  (zkCallback-1748-thread-1-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1772902 INFO  (zkCallback-1746-thread-2-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1772902 INFO  (zkCallback-1750-thread-3-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1772902 INFO  (zkCallback-1750-thread-2-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1772902 INFO  (zkCallback-1746-thread-1-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773284 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1773284 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1773284 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44301/solr/.system_shard1_replica_n1/
   [junit4]   2> 1773284 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=.system_shard1_replica_n1 url=http://127.0.0.1:44301/solr START replicas=[http://127.0.0.1:39455/solr/.system_shard1_replica_n2/] nUpdates=100
   [junit4]   2> 1773288 INFO  (qtp414067309-5708) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.c.S.Request [.system_shard1_replica_n2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1773584 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=.system_shard1_replica_n1 url=http://127.0.0.1:44301/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1773585 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1773585 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1773585 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1773587 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44301/solr/.system_shard1_replica_n1/ shard1
   [junit4]   2> 1773688 INFO  (zkCallback-1748-thread-1-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773688 INFO  (zkCallback-1750-thread-2-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773688 INFO  (zkCallback-1746-thread-1-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773688 INFO  (zkCallback-1748-thread-2-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773688 INFO  (zkCallback-1746-thread-2-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773688 INFO  (zkCallback-1750-thread-3-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773710 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.IndexFetcher Replica core_node3 is leader but it's state is down, skipping replication
   [junit4]   2> 1773738 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1773739 INFO  (qtp396500297-5702) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n1&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2274
   [junit4]   2> 1773803 INFO  (qtp141082257-5695) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=.system&newCollection=true&name=.system_shard1_replica_p4&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=PULL} status=0 QTime=2338
   [junit4]   2> 1773841 INFO  (qtp414067309-5704) [n:127.0.0.1:39455_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n2&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2375
   [junit4]   2> 1773844 INFO  (qtp141082257-5701) [n:127.0.0.1:41593_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1773942 INFO  (zkCallback-1748-thread-1-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773942 INFO  (zkCallback-1746-thread-2-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773942 INFO  (zkCallback-1746-thread-1-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773942 INFO  (zkCallback-1750-thread-3-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773942 INFO  (zkCallback-1750-thread-2-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1773942 INFO  (zkCallback-1748-thread-2-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1774844 INFO  (qtp141082257-5701) [n:127.0.0.1:41593_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=1&replicationFactor=2&name=.system&nrtReplicas=2&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=3703
   [junit4]   2> 1774847 INFO  (qtp141082257-5714) [n:127.0.0.1:41593_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1774849 INFO  (qtp396500297-5709) [n:127.0.0.1:44301_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1774852 INFO  (qtp414067309-5712) [n:127.0.0.1:39455_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 1775128 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAddRemoveNode
   [junit4]   2> 1775129 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1775133 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7c2bcc83{/solr,null,AVAILABLE}
   [junit4]   2> 1775133 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3e5e313e{HTTP/1.1,[http/1.1]}{127.0.0.1:34082}
   [junit4]   2> 1775133 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.e.j.s.Server Started @1790562ms
   [junit4]   2> 1775133 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34082}
   [junit4]   2> 1775133 ERROR (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1775133 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1775134 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1775134 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1775134 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-14T03:36:23.388Z
   [junit4]   2> 1775143 INFO  (zkConnectionManagerCallback-1765-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1775144 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1775162 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35600/solr
   [junit4]   2> 1775163 INFO  (OverseerCollectionConfigSetProcessor-99346086519898121-127.0.0.1:41593_solr-n_0000000000) [n:127.0.0.1:41593_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1775236 INFO  (zkConnectionManagerCallback-1769-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1775349 INFO  (zkConnectionManagerCallback-1771-thread-1-processing-n:127.0.0.1:34082_solr) [n:127.0.0.1:34082_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1775353 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [n:127.0.0.1:34082_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1775354 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [n:127.0.0.1:34082_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1775356 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [n:127.0.0.1:34082_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34082_solr
   [junit4]   2> 1775356 INFO  (zkCallback-1748-thread-2-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1775356 INFO  (zkCallback-1750-thread-2-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1775357 INFO  (zkCallback-1770-thread-1-processing-n:127.0.0.1:34082_solr) [n:127.0.0.1:34082_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1775369 INFO  (zkCallback-1746-thread-1-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1775369 INFO  (zkCallback-1762-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1775480 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [n:127.0.0.1:34082_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34082.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1775500 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [n:127.0.0.1:34082_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34082.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1775500 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [n:127.0.0.1:34082_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34082.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56bb7943
   [junit4]   2> 1775501 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [n:127.0.0.1:34082_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestClusterStateProvider_456CF5C9CB746056-001/tempDir-001/node4/.
   [junit4]   2> 1775799 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
   [junit4]   2> 1775799 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.IndexFetcher Updated masterUrl to http://127.0.0.1:44301/solr/.system_shard1_replica_n1/
   [junit4]   2> 1775800 INFO  (qtp396500297-5713) [n:127.0.0.1:44301_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.S.Request [.system_shard1_replica_n1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1775804 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 1775804 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 1775805 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1775805 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 1775805 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 1775812 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1775812 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2faff5e4 commitCommandVersion:0
   [junit4]   2> 1775829 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.s.SolrIndexSearcher Opening [Searcher@fe63073[.system_shard1_replica_p4] main]
   [junit4]   2> 1775829 INFO  (indexFetcher-2152-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1775926 INFO  (searcherExecutor-2141-thread-1-processing-n:127.0.0.1:41593_solr x:.system_shard1_replica_p4 s:shard1 c:.system r:core_node6) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.c.SolrCore [.system_shard1_replica_p4] Registered new searcher Searcher@fe63073[.system_shard1_replica_p4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1775940 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@161340b1{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1775941 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1159489878
   [junit4]   2> 1775941 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1775941 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4e15b0a6: rootName = solr_41593, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@6c13995
   [junit4]   2> 1775947 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1775947 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7c52ea2: rootName = solr_41593, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@14848a6b
   [junit4]   2> 1775963 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1775963 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6941b02a: rootName = solr_41593, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@48cd7e6a
   [junit4]   2> 1775964 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:41593_solr
   [junit4]   2> 1775985 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1775985 INFO  (zkCallback-1770-thread-1-processing-n:127.0.0.1:34082_solr) [n:127.0.0.1:34082_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1775986 INFO  (zkCallback-1748-thread-2-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1775986 INFO  (zkCallback-1746-thread-1-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1775986 INFO  (zkCallback-1750-thread-2-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1775987 INFO  (zkCallback-1748-thread-1-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1775987 INFO  (zkCallback-1750-thread-3-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1775987 INFO  (zkCallback-1750-thread-1-processing-n:127.0.0.1:41593_solr) [n:127.0.0.1:41593_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1775987 INFO  (zkCallback-1746-thread-2-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1776006 INFO  (zkCallback-1748-thread-3-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1776006 INFO  (coreCloseExecutor-2166-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.c.SolrCore [.system_shard1_replica_p4]  CLOSING SolrCore org.apache.solr.core.SolrCore@475486b7
   [junit4]   2> 1776006 INFO  (coreCloseExecutor-2166-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core..system.shard1.replica_p4, tag=1196721847
   [junit4]   2> 1776007 INFO  (coreCloseExecutor-2166-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5b5a9d70: rootName = solr_41593, domain = solr.core..system.shard1.replica_p4, service url = null, agent id = null] for registry solr.core..system.shard1.replica_p4 / com.codahale.metrics.MetricRegistry@2a2ea5b4
   [junit4]   2> 1776013 INFO  (coreCloseExecutor-2166-thread-1) [n:127.0.0.1:41593_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_p4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection..system.shard1.leader, tag=1196721847
   [junit4]   2> 1776038 INFO  (zkCallback-1746-thread-3-processing-n:127.0.0.1:44301_solr) [n:127.0.0.1:44301_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1776022 INFO  (zkCallback-1762-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 1776042 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.c.Overseer Overseer (id=99346086519898121-127.0.0.1:41593_solr-n_0000000000) closing
   [junit4]   2> 1776043 WARN  (OverseerAutoScalingTriggerThread-99346086519898121-127.0.0.1:41593_solr-n_0000000000) [n:127.0.0.1:41593_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1776100 INFO  (OverseerStateUpdate-99346086519898121-127.0.0.1:41593_solr-n_0000000000) [n:127.0.0.1:41593_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:41593_solr
   [junit4]   2> 1776121 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1b286bf2{/solr,null,UNAVAILABLE}
   [junit4]   2> 1776131 INFO  (zkCallback-1748-thread-3-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39455_solr
   [junit4]   2> 1776153 INFO  (zkCallback-1748-thread-3-processing-n:127.0.0.1:39455_solr) [n:127.0.0.1:39455_solr    ] o.a.s.c.Overseer Overseer (id=99346086519898119-127.0.0.1:39455_solr-n_0000000001) starting
   [junit4]   2> 1776376 INFO  (TEST-TestClusterStateProvider.testAddRemoveNode-seed#[456CF5C9CB746056]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testAddRemoveNode
   [junit4]   2> 1776464 INFO  (TEST-TestClusterStateProvider.testAutoScalingConfig-seed#[456CF5C9CB746056]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAutoScalingConfig
   [junit4]   2> 1776467 INFO  (TEST-TestClusterStateProvider.testAutoScalingConfig-seed#[456CF5C9CB746056]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testAutoScalingConfig
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestClusterStateProvider -Dtests.method=testAutoScalingConfig -Dtests.seed=456CF5C9CB746056 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-VE -Dtests.timezone=America/Hermosillo -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.09s J2 | TestClusterStateProvider.testAutoScalingConfig <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected: org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig<{
   [junit4]    >   "cluster-preferences":[{"maximize":"freedisk"}],
   [junit4]    >   "triggers":{".auto_add_replicas":{
   [junit4]    >       "name":".auto_add_replicas",
   [junit4]    >       "event":"nodeLost",
   [junit4]    >       "waitFor":30,
   [junit4]    >       "actions":[
   [junit4]    >         {
   [junit4]    >           "name":"auto_add_replicas_plan",
   [junit4]    >           "class":"solr.AutoAddReplicasPlanAction"},
   [junit4]    >         {
   [junit4]    >           "name":"execute_plan",
   [junit4]    >           "class":"solr.ExecutePlanAction"}],
   [junit4]    >       "enabled":true}},
   [junit4]    >   "listeners":{".auto_add_replicas.system":{
   [junit4]    >       "trigger":".auto_add_replicas",
   [junit4]    >       "afterAction":[],
   [junit4]    >       "stage":[
   [junit4]    >         "STARTED",
   [junit4]    >         "ABORTED",
   [junit4]    >         "SUCCEEDED",
   [junit4]    >         "FAILED",
   [junit4]    >         "BEFORE_ACTION",
   [junit4]    >         "AFTER_ACTION",
   [junit4]    >         "IGNORED"],
   [junit4]    >       "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",
   [junit4]    >       "beforeAction":[]}},
   [junit4]    >   "properties":{}}> but was: org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig<{
   [junit4]    >   "cluster-preferences":[{"maximize":"freedisk"}],
   [junit4]    >   "triggers":{".auto_add_replicas":{
   [junit4]    >       "name":".auto_add_replicas",
   [junit4]    >       "event":"nodeLost",
   [junit4]    >       "waitFor":30,
   [junit4]    >       "actions":[
   [junit4]    >         {
   [junit4]    >           "name":"auto_add_replicas_plan",
   [junit4]    >           "class":"solr.AutoAddReplicasPlanAction"},
   [junit4]    >         {
   [junit4]    >           "name":"execute_plan",
   [junit4]    >           "class":"solr.ExecutePlanAction"}],
   [junit4]    >       "enabled":true}},
   [junit4]    >   "listeners":{".auto_add_replicas.system":{
   [junit4]    >       "trigger":".auto_add_replicas",
   [junit4]    >       "afterAction":[],
   [junit4]    >       "stage":[
   [junit4]    >         "STARTED",
   [junit4]    >         "ABORTED",
   [junit4]    >         "SUCCEEDED",
   [junit4]    >         "FAILED",
   [junit4]    >         "BEFORE_ACTION",
   [junit4]    >         "AFTER_ACTION",
   [junit4]    >         "IGNORED"],
   [junit4]    >       "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",
   [junit4]    >       "beforeAction":[]}},
   [junit4]    >   "properties":{}}>
   [junit4]    > at __randomizedtesting.SeedInfo.seed([456CF5C9CB746056:7AE4F661DC2290B1]:0)
   [junit4]    > at org.apache.solr.cloud.autoscaling.sim.TestClusterStateProvider.testAutoScalingConfig(TestClusterStateProvider.java:214)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1776483 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1776504 INFO  (jetty-closer-1725-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@510511b2{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1776504 INFO  (jetty-closer-1725-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2073998139
   [junit4]   2> 1776505 INFO  (jetty-closer-1725-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1776505 INFO  (jetty-closer-1725-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.

[...truncated too long message...]

tReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.SolrTestCaseJ4 ###Ending doTestStopPoll
   [junit4]   2> 4518426 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@7702dd43{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 4518426 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1182812762
   [junit4]   2> 4518427 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 4518427 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@61b6ed80: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@46a7f77e
   [junit4]   2> 4518442 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 4518442 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@78c8f6ef: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@35217509
   [junit4]   2> 4518445 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 4518445 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@af51eba: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5176300f
   [junit4]   2> 4518499 INFO  (coreCloseExecutor-6533-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@38a0351e
   [junit4]   2> 4518499 INFO  (coreCloseExecutor-6533-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=950023454
   [junit4]   2> 4518499 INFO  (coreCloseExecutor-6533-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@19263a8b: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@4ce5afa1
   [junit4]   2> 4518651 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4fea49d4{/solr,null,UNAVAILABLE}
   [junit4]   2> 4518734 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3367bd36{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 4518734 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=723761046
   [junit4]   2> 4518750 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 4518750 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@f1cece9: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@5dc2932f
   [junit4]   2> 4518808 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 4518808 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@32637930: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@35217509
   [junit4]   2> 4518845 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 4518845 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@751fd5a6: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5176300f
   [junit4]   2> 4518901 INFO  (coreCloseExecutor-6535-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@bd9d4d5
   [junit4]   2> 4518921 INFO  (coreCloseExecutor-6535-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=198825173
   [junit4]   2> 4518921 INFO  (coreCloseExecutor-6535-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@56e75e4c: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@cd08059
   [junit4]   2> 4519067 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[456CF5C9CB746056]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7f89410e{/solr,null,UNAVAILABLE}
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_456CF5C9CB746056-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name=FST50, id=FST50, newname=FST50}, docValues:{}, maxPointsInLeafNode=1131, maxMBSortInHeap=7.701995861154797, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@755b69e8), locale=es-UY, timezone=Europe/Prague
   [junit4]   2> NOTE: Linux 4.4.0-104-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=313930328,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [TestZkAclsWithHadoopAuth, TestSubQueryTransformerDistrib, BadCopyFieldTest, PathHierarchyTokenizerFactoryTest, TestGraphTermsQParserPlugin, OutputWriterTest, TestRemoteStreaming, MoveReplicaTest, SpatialHeatmapFacetsTest, BigEndianAscendingWordSerializerTest, TestPayloadCheckQParserPlugin, LeaderFailoverAfterPartitionTest, DistributedDebugComponentTest, TestSolr4Spatial, StressHdfsTest, TestRTimerTree, TestPKIAuthenticationPlugin, TestPullReplicaErrorHandling, SimpleCollectionCreateDeleteTest, TestFieldCacheWithThreads, TestCloudInspectUtil, TestCryptoKeys, SolrJmxReporterTest, TestDocumentBuilder, TestJmxIntegration, ShardRoutingTest, TestRandomCollapseQParserPlugin, TestScoreJoinQPScore, TestStressCloudBlindAtomicUpdates, TestUseDocValuesAsStored2, TestDownShardTolerantSearch, DistributedVersionInfoTest, TestSimpleTextCodec, TestReqParamsAPI, ConvertedLegacyTest, DistributedQueryComponentOptimizationTest, TestClusterProperties, TestBulkSchemaConcurrent, TestExceedMaxTermLength, CachingDirectoryFactoryTest, UtilsToolTest, VMParamsZkACLAndCredentialsProvidersTest, PropertiesRequestHandlerTest, NumberUtilsTest, TestCloudPseudoReturnFields, TestSchemaManager, TestNRTOpen, TestBlendedInfixSuggestions, TestCollapseQParserPlugin, DocValuesMultiTest, OpenCloseCoreStressTest, SparseHLLTest, ConjunctionSolrSpellCheckerTest, TestOmitPositions, HdfsChaosMonkeyNothingIsSafeTest, TestSimpleQParserPlugin, BadIndexSchemaTest, TestMacroExpander, CloneFieldUpdateProcessorFactoryTest, TestLazyCores, RulesTest, PolyFieldTest, CoreMergeIndexesAdminHandlerTest, DirectUpdateHandlerTest, TestConfigSetImmutable, HealthCheckHandlerTest, AutoAddReplicasPlanActionTest, TestJavabinTupleStreamParser, DeleteShardTest, PreAnalyzedFieldTest, SmileWriterTest, TestSolrQueryParser, DocumentAnalysisRequestHandlerTest, NumericFieldsTest, ChaosMonkeySafeLeaderWithPullReplicasTest, TestPerFieldSimilarity, LukeRequestHandlerTest, CreateCollectionCleanupTest, TestSolrCoreParser, TestImplicitCoreProperties, PeerSyncWithIndexFingerprintCachingTest, PingRequestHandlerTest, CloudExitableDirectoryReaderTest, ExecutePlanActionTest, ConcurrentDeleteAndCreateCollectionTest, CircularListTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestSimpleTrackingShardHandler, ZkSolrClientTest, TestUtilizeNode, TestCloudNestedDocsSort, SearchRateTriggerTest, HighlighterMaxOffsetTest, TestConfigsApi, SolrGangliaReporterTest, TestRTGBase, LeaderInitiatedRecoveryOnShardRestartTest, TestFastLRUCache, TestNumericRangeQuery64, SolrIndexSplitterTest, MBeansHandlerTest, TestRangeQuery, TestCollationFieldDocValues, SolrMetricManagerTest, SampleTest, DistributedFacetPivotLargeTest, TestWriterPerf, TestDelegationWithHadoopAuth, DistributedFacetPivotSmallTest, TestExactStatsCache, ClusterStateUpdateTest, MissingSegmentRecoveryTest, TestDocSet, ShardRoutingCustomTest, EnumFieldTest, TolerantUpdateProcessorTest, TestSolrCloudWithDelegationTokens, TestFastWriter, SignatureUpdateProcessorFactoryTest, TestCoreBackup, AssignTest, ZkStateWriterTest, AddReplicaTest, TestStressLucene, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest, BasicZkTest, RecoveryZkTest, TestReplicationHandler]
   [junit4] Completed [401/767 (2!)] on J1 in 422.62s, 16 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 49324 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1449 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1449/

7 tests failed.
FAILED:  org.apache.lucene.index.TestMixedDocValuesUpdates.testTonsOfUpdates

Error Message:
_1_l.fnm in dir=RAMDirectory@e8d59da lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5cdd87eb

Stack Trace:
java.nio.file.NoSuchFileException: _1_l.fnm in dir=RAMDirectory@e8d59da lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5cdd87eb
        at __randomizedtesting.SeedInfo.seed([7D3E6928A520CDD7:51BB7234700E235]:0)
        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:750)
        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:119)
        at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
        at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:113)
        at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:93)
        at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:688)
        at org.apache.lucene.index.IndexWriter$ReaderPool.writeSomeDocValuesUpdates(IndexWriter.java:705)
        at org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:332)
        at org.apache.lucene.index.DocumentsWriter$ResolveUpdatesEvent.process(DocumentsWriter.java:740)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5104)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5094)
        at org.apache.lucene.index.IndexWriter.updateDocValues(IndexWriter.java:1889)
        at org.apache.lucene.index.TestMixedDocValuesUpdates.testTonsOfUpdates(TestMixedDocValuesUpdates.java:374)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.RestartWhileUpdatingTest.test

Error Message:
shard1 is not consistent.  Got 1007 from https://127.0.0.1:46010/y/hi/collection1_shard1_replica_n21 (previous client) and got 1248 from https://127.0.0.1:44643/y/hi/collection1_shard1_replica_n23

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 1007 from https://127.0.0.1:46010/y/hi/collection1_shard1_replica_n21 (previous client) and got 1248 from https://127.0.0.1:44643/y/hi/collection1_shard1_replica_n23
        at __randomizedtesting.SeedInfo.seed([6188612105A09390:E9DC5EFBAB5CFE68]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1328)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1307)
        at org.apache.solr.cloud.RestartWhileUpdatingTest.test(RestartWhileUpdatingTest.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RestartWhileUpdatingTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.RestartWhileUpdatingTest:     1) Thread[id=12319, name=searcherExecutor-2251-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.RestartWhileUpdatingTest:
   1) Thread[id=12319, name=searcherExecutor-2251-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([6188612105A09390]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RestartWhileUpdatingTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=12319, name=searcherExecutor-2251-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=12319, name=searcherExecutor-2251-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([6188612105A09390]:0)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testListeners

Error Message:
foo events: [CapturedEvent{timestamp=452002502401546, stage=STARTED, actionName='null', event={   "id":"19b178d181344Tdlh6rgdzkoyndpp6uu9gvesbb",   "source":"node_added_trigger",   "eventTime":452000430428996,   "eventType":"NODEADDED",   "properties":{     "eventTimes":[452000430428996],     "_enqueue_time_":452002483050696,     "nodeNames":["127.0.0.1:10013_solr"]}}, context={}, config={   "trigger":"node_added_trigger",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "afterAction":[     "test",     "test1"],   "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",   "beforeAction":"test"}, message='null'}, CapturedEvent{timestamp=452002509401596, stage=BEFORE_ACTION, actionName='test', event={   "id":"19b178d181344Tdlh6rgdzkoyndpp6uu9gvesbb",   "source":"node_added_trigger",   "eventTime":452000430428996,   "eventType":"NODEADDED",   "properties":{     "eventTimes":[452000430428996],     "_enqueue_time_":452002483050696,     "nodeNames":["127.0.0.1:10013_solr"]}}, context={properties.BEFORE_ACTION=[test], source=node_added_trigger}, config={   "trigger":"node_added_trigger",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "afterAction":[     "test",     "test1"],   "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",   "beforeAction":"test"}, message='null'}, CapturedEvent{timestamp=452002511149446, stage=AFTER_ACTION, actionName='test', event={   "id":"19b178d181344Tdlh6rgdzkoyndpp6uu9gvesbb",   "source":"node_added_trigger",   "eventTime":452000430428996,   "eventType":"NODEADDED",   "properties":{     "eventTimes":[452000430428996],     "_enqueue_time_":452002483050696,     "nodeNames":["127.0.0.1:10013_solr"]}}, context={properties.BEFORE_ACTION=[test], source=node_added_trigger, properties.AFTER_ACTION=[test]}, config={   "trigger":"node_added_trigger",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "afterAction":[     "test",     "test1"],   "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",   "beforeAction":"test"}, message='null'}] expected:<4> but was:<3>

Stack Trace:
java.lang.AssertionError: foo events: [CapturedEvent{timestamp=452002502401546, stage=STARTED, actionName='null', event={
  "id":"19b178d181344Tdlh6rgdzkoyndpp6uu9gvesbb",
  "source":"node_added_trigger",
  "eventTime":452000430428996,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[452000430428996],
    "_enqueue_time_":452002483050696,
    "nodeNames":["127.0.0.1:10013_solr"]}}, context={}, config={
  "trigger":"node_added_trigger",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "afterAction":[
    "test",
    "test1"],
  "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
  "beforeAction":"test"}, message='null'}, CapturedEvent{timestamp=452002509401596, stage=BEFORE_ACTION, actionName='test', event={
  "id":"19b178d181344Tdlh6rgdzkoyndpp6uu9gvesbb",
  "source":"node_added_trigger",
  "eventTime":452000430428996,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[452000430428996],
    "_enqueue_time_":452002483050696,
    "nodeNames":["127.0.0.1:10013_solr"]}}, context={properties.BEFORE_ACTION=[test], source=node_added_trigger}, config={
  "trigger":"node_added_trigger",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "afterAction":[
    "test",
    "test1"],
  "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
  "beforeAction":"test"}, message='null'}, CapturedEvent{timestamp=452002511149446, stage=AFTER_ACTION, actionName='test', event={
  "id":"19b178d181344Tdlh6rgdzkoyndpp6uu9gvesbb",
  "source":"node_added_trigger",
  "eventTime":452000430428996,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[452000430428996],
    "_enqueue_time_":452002483050696,
    "nodeNames":["127.0.0.1:10013_solr"]}}, context={properties.BEFORE_ACTION=[test], source=node_added_trigger, properties.AFTER_ACTION=[test]}, config={
  "trigger":"node_added_trigger",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "afterAction":[
    "test",
    "test1"],
  "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
  "beforeAction":"test"}, message='null'}] expected:<4> but was:<3>
        at __randomizedtesting.SeedInfo.seed([6188612105A09390:8C48F631B8CC0E49]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testListeners(TestTriggerIntegration.java:1009)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestClusterStateProvider.testAutoScalingConfig

Error Message:
expected: org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig<{   "cluster-preferences":[{"maximize":"freedisk"}],   "triggers":{".auto_add_replicas":{       "name":".auto_add_replicas",       "event":"nodeLost",       "waitFor":30,       "actions":[         {           "name":"auto_add_replicas_plan",           "class":"solr.AutoAddReplicasPlanAction"},         {           "name":"execute_plan",           "class":"solr.ExecutePlanAction"}],       "enabled":true}},   "listeners":{".auto_add_replicas.system":{       "trigger":".auto_add_replicas",       "afterAction":[],       "stage":[         "STARTED",         "ABORTED",         "SUCCEEDED",         "FAILED",         "BEFORE_ACTION",         "AFTER_ACTION",         "IGNORED"],       "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",       "beforeAction":[]}},   "properties":{}}> but was: org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig<{   "cluster-preferences":[{"maximize":"freedisk"}],   "triggers":{".auto_add_replicas":{       "name":".auto_add_replicas",       "event":"nodeLost",       "waitFor":30,       "actions":[         {           "name":"auto_add_replicas_plan",           "class":"solr.AutoAddReplicasPlanAction"},         {           "name":"execute_plan",           "class":"solr.ExecutePlanAction"}],       "enabled":true}},   "listeners":{".auto_add_replicas.system":{       "trigger":".auto_add_replicas",       "afterAction":[],       "stage":[         "STARTED",         "ABORTED",         "SUCCEEDED",         "FAILED",         "BEFORE_ACTION",         "AFTER_ACTION",         "IGNORED"],       "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",       "beforeAction":[]}},   "properties":{}}>

Stack Trace:
java.lang.AssertionError: expected: org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig<{
  "cluster-preferences":[{"maximize":"freedisk"}],
  "triggers":{".auto_add_replicas":{
      "name":".auto_add_replicas",
      "event":"nodeLost",
      "waitFor":30,
      "actions":[
        {
          "name":"auto_add_replicas_plan",
          "class":"solr.AutoAddReplicasPlanAction"},
        {
          "name":"execute_plan",
          "class":"solr.ExecutePlanAction"}],
      "enabled":true}},
  "listeners":{".auto_add_replicas.system":{
      "trigger":".auto_add_replicas",
      "afterAction":[],
      "stage":[
        "STARTED",
        "ABORTED",
        "SUCCEEDED",
        "FAILED",
        "BEFORE_ACTION",
        "AFTER_ACTION",
        "IGNORED"],
      "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",
      "beforeAction":[]}},
  "properties":{}}> but was: org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig<{
  "cluster-preferences":[{"maximize":"freedisk"}],
  "triggers":{".auto_add_replicas":{
      "name":".auto_add_replicas",
      "event":"nodeLost",
      "waitFor":30,
      "actions":[
        {
          "name":"auto_add_replicas_plan",
          "class":"solr.AutoAddReplicasPlanAction"},
        {
          "name":"execute_plan",
          "class":"solr.ExecutePlanAction"}],
      "enabled":true}},
  "listeners":{".auto_add_replicas.system":{
      "trigger":".auto_add_replicas",
      "afterAction":[],
      "stage":[
        "STARTED",
        "ABORTED",
        "SUCCEEDED",
        "FAILED",
        "BEFORE_ACTION",
        "AFTER_ACTION",
        "IGNORED"],
      "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",
      "beforeAction":[]}},
  "properties":{}}>
        at __randomizedtesting.SeedInfo.seed([6188612105A09390:5E00628912F66377]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.cloud.autoscaling.sim.TestClusterStateProvider.testAutoScalingConfig(TestClusterStateProvider.java:214)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1516002348100,generation=2,filelist=[_6u8.fdt, _6u8.fdx, _6u8.fnm, _6u8.nvd, _6u8.nvm, _6u8.si, _6u8_Memory_0.ram, _7tf.fdt, _7tf.fdx, _7tf.fnm, _7tf.nvd, _7tf.nvm, _7tf.si, _7tf_Memory_0.ram, _9cs.fdt, _9cs.fdx, _9cs.fnm, _9cs.nvd, _9cs.nvm, _9cs.si, _9cs_Memory_0.ram, _d4x.fdt, _d4x.fdx, _d4x.fnm, _d4x.nvd, _d4x.nvm, _d4x.si, _d4x_Memory_0.ram, _e66.cfe, _e66.cfs, _e66.si, _e9t.cfe, _e9t.cfs, _e9t.si, _ea2.cfe, _ea2.cfs, _ea2.si, _ea3.cfe, _ea3.cfs, _ea3.si, _ea4.cfe, _ea4.cfs, _ea4.si, _ea5.cfe, _ea5.cfs, _ea5.si, _ea6.cfe, _ea6.cfs, _ea6.si, _ea7.cfe, _ea7.cfs, _ea7.si, _ea8.cfe, _ea8.cfs, _ea8.si, _ea9.cfe, _ea9.cfs, _ea9.si, _eaa.cfe, _eaa.cfs, _eaa.si, _eab.cfe, _eab.cfs, _eab.si, _ead.cfe, _ead.cfs, _ead.si, segments_2]}]> but was:<[{indexVersion=1516002348100,generation=2,filelist=[_6u8.fdt, _6u8.fdx, _6u8.fnm, _6u8.nvd, _6u8.nvm, _6u8.si, _6u8_Memory_0.ram, _7tf.fdt, _7tf.fdx, _7tf.fnm, _7tf.nvd, _7tf.nvm, _7tf.si, _7tf_Memory_0.ram, _9cs.fdt, _9cs.fdx, _9cs.fnm, _9cs.nvd, _9cs.nvm, _9cs.si, _9cs_Memory_0.ram, _d4x.fdt, _d4x.fdx, _d4x.fnm, _d4x.nvd, _d4x.nvm, _d4x.si, _d4x_Memory_0.ram, _e66.cfe, _e66.cfs, _e66.si, _e9t.cfe, _e9t.cfs, _e9t.si, _ea2.cfe, _ea2.cfs, _ea2.si, _ea3.cfe, _ea3.cfs, _ea3.si, _ea4.cfe, _ea4.cfs, _ea4.si, _ea5.cfe, _ea5.cfs, _ea5.si, _ea6.cfe, _ea6.cfs, _ea6.si, _ea7.cfe, _ea7.cfs, _ea7.si, _ea8.cfe, _ea8.cfs, _ea8.si, _ea9.cfe, _ea9.cfs, _ea9.si, _eaa.cfe, _eaa.cfs, _eaa.si, _eab.cfe, _eab.cfs, _eab.si, _ead.cfe, _ead.cfs, _ead.si, segments_2]}, {indexVersion=1516002348100,generation=3,filelist=[_6u8.fdt, _6u8.fdx, _6u8.fnm, _6u8.nvd, _6u8.nvm, _6u8.si, _6u8_Memory_0.ram, _7tf.fdt, _7tf.fdx, _7tf.fnm, _7tf.nvd, _7tf.nvm, _7tf.si, _7tf_Memory_0.ram, _9cs.fdt, _9cs.fdx, _9cs.fnm, _9cs.nvd, _9cs.nvm, _9cs.si, _9cs_Memory_0.ram, _d4x.fdt, _d4x.fdx, _d4x.fnm, _d4x.nvd, _d4x.nvm, _d4x.si, _d4x_Memory_0.ram, _e66.cfe, _e66.cfs, _e66.si, _e9t.cfe, _e9t.cfs, _e9t.si, _eac.cfe, _eac.cfs, _eac.si, _ead.cfe, _ead.cfs, _ead.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1516002348100,generation=2,filelist=[_6u8.fdt, _6u8.fdx, _6u8.fnm, _6u8.nvd, _6u8.nvm, _6u8.si, _6u8_Memory_0.ram, _7tf.fdt, _7tf.fdx, _7tf.fnm, _7tf.nvd, _7tf.nvm, _7tf.si, _7tf_Memory_0.ram, _9cs.fdt, _9cs.fdx, _9cs.fnm, _9cs.nvd, _9cs.nvm, _9cs.si, _9cs_Memory_0.ram, _d4x.fdt, _d4x.fdx, _d4x.fnm, _d4x.nvd, _d4x.nvm, _d4x.si, _d4x_Memory_0.ram, _e66.cfe, _e66.cfs, _e66.si, _e9t.cfe, _e9t.cfs, _e9t.si, _ea2.cfe, _ea2.cfs, _ea2.si, _ea3.cfe, _ea3.cfs, _ea3.si, _ea4.cfe, _ea4.cfs, _ea4.si, _ea5.cfe, _ea5.cfs, _ea5.si, _ea6.cfe, _ea6.cfs, _ea6.si, _ea7.cfe, _ea7.cfs, _ea7.si, _ea8.cfe, _ea8.cfs, _ea8.si, _ea9.cfe, _ea9.cfs, _ea9.si, _eaa.cfe, _eaa.cfs, _eaa.si, _eab.cfe, _eab.cfs, _eab.si, _ead.cfe, _ead.cfs, _ead.si, segments_2]}]> but was:<[{indexVersion=1516002348100,generation=2,filelist=[_6u8.fdt, _6u8.fdx, _6u8.fnm, _6u8.nvd, _6u8.nvm, _6u8.si, _6u8_Memory_0.ram, _7tf.fdt, _7tf.fdx, _7tf.fnm, _7tf.nvd, _7tf.nvm, _7tf.si, _7tf_Memory_0.ram, _9cs.fdt, _9cs.fdx, _9cs.fnm, _9cs.nvd, _9cs.nvm, _9cs.si, _9cs_Memory_0.ram, _d4x.fdt, _d4x.fdx, _d4x.fnm, _d4x.nvd, _d4x.nvm, _d4x.si, _d4x_Memory_0.ram, _e66.cfe, _e66.cfs, _e66.si, _e9t.cfe, _e9t.cfs, _e9t.si, _ea2.cfe, _ea2.cfs, _ea2.si, _ea3.cfe, _ea3.cfs, _ea3.si, _ea4.cfe, _ea4.cfs, _ea4.si, _ea5.cfe, _ea5.cfs, _ea5.si, _ea6.cfe, _ea6.cfs, _ea6.si, _ea7.cfe, _ea7.cfs, _ea7.si, _ea8.cfe, _ea8.cfs, _ea8.si, _ea9.cfe, _ea9.cfs, _ea9.si, _eaa.cfe, _eaa.cfs, _eaa.si, _eab.cfe, _eab.cfs, _eab.si, _ead.cfe, _ead.cfs, _ead.si, segments_2]}, {indexVersion=1516002348100,generation=3,filelist=[_6u8.fdt, _6u8.fdx, _6u8.fnm, _6u8.nvd, _6u8.nvm, _6u8.si, _6u8_Memory_0.ram, _7tf.fdt, _7tf.fdx, _7tf.fnm, _7tf.nvd, _7tf.nvm, _7tf.si, _7tf_Memory_0.ram, _9cs.fdt, _9cs.fdx, _9cs.fnm, _9cs.nvd, _9cs.nvm, _9cs.si, _9cs_Memory_0.ram, _d4x.fdt, _d4x.fdx, _d4x.fnm, _d4x.nvd, _d4x.nvm, _d4x.si, _d4x_Memory_0.ram, _e66.cfe, _e66.cfs, _e66.si, _e9t.cfe, _e9t.cfs, _e9t.si, _eac.cfe, _eac.cfs, _eac.si, _ead.cfe, _ead.cfs, _ead.si, segments_3]}]>
        at __randomizedtesting.SeedInfo.seed([6188612105A09390:445F7A1175E89D93]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1277)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 456 lines...]
   [junit4] Suite: org.apache.lucene.index.TestMixedDocValuesUpdates
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestMixedDocValuesUpdates -Dtests.method=testTonsOfUpdates -Dtests.seed=7D3E6928A520CDD7 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-ES -Dtests.timezone=Etc/UTC -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   23.4s J2 | TestMixedDocValuesUpdates.testTonsOfUpdates <<<
   [junit4]    > Throwable #1: java.nio.file.NoSuchFileException: _1_l.fnm in dir=RAMDirectory@e8d59da lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5cdd87eb
   [junit4]    > at __randomizedtesting.SeedInfo.seed([7D3E6928A520CDD7:51BB7234700E235]:0)
   [junit4]    > at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:750)
   [junit4]    > at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:119)
   [junit4]    > at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
   [junit4]    > at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:113)
   [junit4]    > at org.apache.lucene.index.SegmentReader.initFieldInfos(SegmentReader.java:190)
   [junit4]    > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:93)
   [junit4]    > at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:688)
   [junit4]    > at org.apache.lucene.index.IndexWriter$ReaderPool.writeSomeDocValuesUpdates(IndexWriter.java:705)
   [junit4]    > at org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:332)
   [junit4]    > at org.apache.lucene.index.DocumentsWriter$ResolveUpdatesEvent.process(DocumentsWriter.java:740)
   [junit4]    > at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5104)
   [junit4]    > at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5094)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocValues(IndexWriter.java:1889)
   [junit4]    > at org.apache.lucene.index.TestMixedDocValuesUpdates.testTonsOfUpdates(TestMixedDocValuesUpdates.java:374)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestMixedDocValuesUpdates_7D3E6928A520CDD7-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3299b0f7), locale=es-ES, timezone=Etc/UTC
   [junit4]   2> NOTE: Linux 4.4.0-104-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=118090952,total=368574464
   [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, TestConsistentFieldNumbers, TestCompiledAutomaton, TestAssertions, TestTermScorer, TestBKD, TestStopFilter, TestIntroSelector, TestDoubleValuesSource, TestRecyclingIntBlockAllocator, TestRoaringDocIdSet, Test2BPostingsBytes, TestTermRangeQuery, TestShardSearching, TestIndexCommit, TestLucene60PointsFormat, TestSparseFixedBitDocIdSet, TestNRTThreads, TestReqExclBulkScorer, TestSpanSearchEquivalence, TestWANDScorer, TestBlockPostingsFormat, TestBooleanScorer, TestMixedDocValuesUpdates]
   [junit4] Completed [65/476 (1!)] on J2 in 31.68s, 4 tests, 1 error <<< FAILURES!

[...truncated 12313 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
   [junit4]   2> 1013372 INFO  (SUITE-TestTriggerIntegration-seed#[6188612105A09390]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_6188612105A09390-001/init-core-data-001
   [junit4]   2> 1013385 WARN  (SUITE-TestTriggerIntegration-seed#[6188612105A09390]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=16 numCloses=16
   [junit4]   2> 1013385 INFO  (SUITE-TestTriggerIntegration-seed#[6188612105A09390]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1013386 INFO  (SUITE-TestTriggerIntegration-seed#[6188612105A09390]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1013425 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1013425 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 1013425 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: []
   [junit4]   2> 1013425 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 1013425 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 1013427 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1013427 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
   [junit4]   2> 1013428 DEBUG (ScheduledTrigger-709-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1013443 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCooldown
   [junit4]   2> 1013443 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1013443 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1013444 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 1013539 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1013539 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1013539 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1013540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 1013540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 1013540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 1013540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 1013540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1013540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 1013540 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1013559 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1013559 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
   [junit4]   2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1013560 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1013591 DEBUG (ScheduledTrigger-712-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1013612 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1013632 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1013656 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1013663 DEBUG (simCloudManagerPool-711-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1013664 DEBUG (simCloudManagerPool-711-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 1013688 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1013690 DEBUG (simCloudManagerPool-711-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 1013690 DEBUG (simCloudManagerPool-711-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
   [junit4]   2> 1013690 DEBUG (simCloudManagerPool-711-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 1013690 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1013690 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1013691 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1013691 DEBUG (ScheduledTrigger-712-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013691 DEBUG (ScheduledTrigger-712-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1013691 DEBUG (simCloudManagerPool-711-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1013691 DEBUG (simCloudManagerPool-711-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 1013691 DEBUG (simCloudManagerPool-711-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 1013691 DEBUG (simCloudManagerPool-711-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
   [junit4]   2> 1013691 DEBUG (simCloudManagerPool-711-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 1013692 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 1013692 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 1013709 DEBUG (ScheduledTrigger-712-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1013712 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013712 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1013712 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10002_solr at time 451836432041346
   [junit4]   2> 1013729 DEBUG (ScheduledTrigger-712-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1013732 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013732 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1013732 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger firing registered processor for nodes: [127.0.0.1:10002_solr] added at times [451836432041346], now=451837455684596
   [junit4]   2> 1013732 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"19af15e074582Tdlh6rgdzkoyndpp6uu9gves9g",
   [junit4]   2>   "source":"node_added_cooldown_trigger",
   [junit4]   2>   "eventTime":451836432041346,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451836432041346],
   [junit4]   2>     "nodeNames":["127.0.0.1:10002_solr"]}}
   [junit4]   2> 1013754 DEBUG (AutoscalingActionExecutor-713-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"19af15e074582Tdlh6rgdzkoyndpp6uu9gves9g",
   [junit4]   2>   "source":"node_added_cooldown_trigger",
   [junit4]   2>   "eventTime":451836432041346,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451836432041346],
   [junit4]   2>     "_enqueue_time_":451837471327596,
   [junit4]   2>     "nodeNames":["127.0.0.1:10002_solr"]}}
   [junit4]   2> 1013756 DEBUG (AutoscalingActionExecutor-713-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 65 ms for event id=19af15e074582Tdlh6rgdzkoyndpp6uu9gves9g
   [junit4]   2> 1013769 DEBUG (ScheduledTrigger-712-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1013775 DEBUG (ScheduledTrigger-712-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013775 DEBUG (ScheduledTrigger-712-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1013790 DEBUG (ScheduledTrigger-712-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 1013795 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013795 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1013795 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10003_solr at time 451840609946996
   [junit4]   2> 1013810 DEBUG (ScheduledTrigger-712-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 1013815 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013815 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1013815 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger firing registered processor for nodes: [127.0.0.1:10003_solr] added at times [451840609946996], now=451841624097596
   [junit4]   2> 1013816 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
   [junit4]   2>   "id":"19af2570d0d74Tdlh6rgdzkoyndpp6uu9gves9j",
   [junit4]   2>   "source":"node_added_cooldown_trigger",
   [junit4]   2>   "eventTime":451840609946996,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451840609946996],
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 1013830 DEBUG (ScheduledTrigger-712-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 1013836 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013836 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1013836 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger firing registered processor for nodes: [127.0.0.1:10003_solr] added at times [451840609946996], now=451842666311246
   [junit4]   2> 1013836 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
   [junit4]   2>   "id":"19af2570d0d74Tdlh6rgdzkoyndpp6uu9gves9l",
   [junit4]   2>   "source":"node_added_cooldown_trigger",
   [junit4]   2>   "eventTime":451840609946996,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451840609946996],
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 1013850 DEBUG (ScheduledTrigger-712-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 1013857 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013857 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1013857 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger firing registered processor for nodes: [127.0.0.1:10003_solr] added at times [451840609946996], now=451843699502796
   [junit4]   2> 1013857 DEBUG (ScheduledTrigger-712-thread-1) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"19af2570d0d74Tdlh6rgdzkoyndpp6uu9gves9n",
   [junit4]   2>   "source":"node_added_cooldown_trigger",
   [junit4]   2>   "eventTime":451840609946996,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451840609946996],
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 1013858 DEBUG (AutoscalingActionExecutor-713-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"19af2570d0d74Tdlh6rgdzkoyndpp6uu9gves9n",
   [junit4]   2>   "source":"node_added_cooldown_trigger",
   [junit4]   2>   "eventTime":451840609946996,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451840609946996],
   [junit4]   2>     "_enqueue_time_":451843706473946,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 1013861 DEBUG (AutoscalingActionExecutor-713-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 196 ms for event id=19af2570d0d74Tdlh6rgdzkoyndpp6uu9gves9n
   [junit4]   2> 1013870 DEBUG (ScheduledTrigger-712-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 1013879 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013879 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1013891 DEBUG (ScheduledTrigger-712-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 1013898 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCooldown
   [junit4]   2> 1013898 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase
   [junit4]   2>
   [junit4]   2> 1013898 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 1013898 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
   [junit4]   2> 1013898 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2>
   [junit4]   2> 1013898 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 4
   [junit4]   2> 1013898 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 4
   [junit4]   2> 1013899 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
   [junit4]   2> 1013899 DEBUG (ScheduledTrigger-712-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013899 DEBUG (ScheduledTrigger-712-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1013899 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
   [junit4]   2> 1013899 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
   [junit4]   2> 1013900 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
   [junit4]   2> 1013900 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
   [junit4]   2> 1013900 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
   [junit4]   2> 1013900 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling        2
   [junit4]   2> 1013900 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update             6
   [junit4]   2> 1013900 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
   [junit4]   2> 1013901 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_cooldown_trigger
   [junit4]   2> 1013901 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - IGNORED         2
   [junit4]   2> 1013901 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED         2
   [junit4]   2> 1013901 INFO  (TEST-TestTriggerIntegration.testCooldown-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED       2
   [junit4]   2> 1013911 DEBUG (ScheduledTrigger-712-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 1013927 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
   [junit4]   2> 1013928 DEBUG (ScheduledTrigger-712-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1013931 INFO  (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[6188612105A09390]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerThrottling
   [junit4]   2> 1013931 DEBUG (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[6188612105A09390]) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 1013931 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 1013931 DEBUG (ScheduledTrigger-712-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
   [junit4]   2> 1013932 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1013932 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1013932 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 1013942 INFO  (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[6188612105A09390]) [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
   [junit4]   2> 1013942 WARN  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
   [junit4]   2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1013943 DEBUG (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[6188612105A09390]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1013957 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1013957 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 1013957 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1013957 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion -1
   [junit4]   2> 1013957 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 1013978 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1013978 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 1013979 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1014013 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1014049 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1014089 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1014102 DEBUG (simCloudManagerPool-714-thread-1) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1014102 DEBUG (simCloudManagerPool-714-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 1014104 DEBUG (simCloudManagerPool-714-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014104 DEBUG (simCloudManagerPool-714-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014104 DEBUG (simCloudManagerPool-714-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1014104 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 1014104 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1014105 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 1014115 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1014115 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1014115 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1014116 DEBUG (simCloudManagerPool-714-thread-2) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1014116 DEBUG (simCloudManagerPool-714-thread-2) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 1014116 DEBUG (simCloudManagerPool-714-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014116 DEBUG (simCloudManagerPool-714-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1014116 DEBUG (simCloudManagerPool-714-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014116 DEBUG (simCloudManagerPool-714-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014116 DEBUG (simCloudManagerPool-714-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1014116 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 1014117 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1014117 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 1014117 DEBUG (ScheduledTrigger-715-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1014117 DEBUG (ScheduledTrigger-715-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1014117 DEBUG (ScheduledTrigger-715-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10004_solr at time 451856690014246
   [junit4]   2> 1014117 DEBUG (ScheduledTrigger-715-thread-4) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:10004_solr] added at times [451856690014246], now=451856691269046
   [junit4]   2> 1014117 DEBUG (ScheduledTrigger-715-thread-4) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"19af6157f6826Tdlh6rgdzkoyndpp6uu9gves9q",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":451856690014246,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451856690014246],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1014117 DEBUG (AutoscalingActionExecutor-716-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"19af6157f6826Tdlh6rgdzkoyndpp6uu9gves9q",
   [junit4]   2>   "source":"node_added_trigger2",
   [junit4]   2>   "eventTime":451856690014246,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451856690014246],
   [junit4]   2>     "_enqueue_time_":451856697838896,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1014117 INFO  (AutoscalingActionExecutor-716-thread-1) [    ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger2
   [junit4]   2> 1014130 DEBUG (AutoscalingActionExecutor-716-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 655 ms for event id=19af6157f6826Tdlh6rgdzkoyndpp6uu9gves9q
   [junit4]   2> 1014159 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1014160 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1014160 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1014160 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10004_solr at time 451858851432046
   [junit4]   2> 1014160 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10004_solr] added at times [451858851432046], now=451858852877246
   [junit4]   2> 1014160 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
   [junit4]   2>   "id":"19af69654066eTdlh6rgdzkoyndpp6uu9gves9t",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":451858851432046,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451858851432046],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1014178 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1014178 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1014179 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1014198 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1014198 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1014198 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1014198 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1014199 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10004_solr] added at times [451858851432046], now=451860777721096
   [junit4]   2> 1014199 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
   [junit4]   2>   "id":"19af69654066eTdlh6rgdzkoyndpp6uu9gves9v",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":451858851432046,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451858851432046],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1014207 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1014219 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1014219 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1014228 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 1014247 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1014248 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1014248 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10004_solr] added at times [451858851432046], now=451863230014446
   [junit4]   2> 1014248 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"19af69654066eTdlh6rgdzkoyndpp6uu9gves9x",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":451858851432046,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451858851432046],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1014248 DEBUG (AutoscalingActionExecutor-716-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"19af69654066eTdlh6rgdzkoyndpp6uu9gves9x",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":451858851432046,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451858851432046],
   [junit4]   2>     "_enqueue_time_":451863237594046,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1014248 INFO  (AutoscalingActionExecutor-716-thread-1) [    ] o.a.s.c.a.s.TestTriggerIntegration last action at 451856726100596 time = 451863268101096
   [junit4]   2> 1014248 INFO  (AutoscalingActionExecutor-716-thread-1) [    ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger1
   [junit4]   2> 1014250 DEBUG (AutoscalingActionExecutor-716-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 77 ms for event id=19af69654066eTdlh6rgdzkoyndpp6uu9gves9x
   [junit4]   2> 1014264 DEBUG (simCloudManagerPool-714-thread-9) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1014264 DEBUG (simCloudManagerPool-714-thread-9) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 1014264 DEBUG (simCloudManagerPool-714-thread-9) [    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014264 DEBUG (simCloudManagerPool-714-thread-9) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1014264 DEBUG (simCloudManagerPool-714-thread-9) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014264 DEBUG (simCloudManagerPool-714-thread-9) [    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014265 DEBUG (simCloudManagerPool-714-thread-9) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1014265 DEBUG (simCloudManagerPool-714-thread-9) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014265 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 1014265 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1014265 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 1014265 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 3
   [junit4]   2> 1014266 DEBUG (simCloudManagerPool-714-thread-10) [    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1014266 DEBUG (simCloudManagerPool-714-thread-10) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
   [junit4]   2> 1014266 DEBUG (simCloudManagerPool-714-thread-10) [    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014266 DEBUG (simCloudManagerPool-714-thread-10) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1014266 DEBUG (simCloudManagerPool-714-thread-10) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014266 DEBUG (simCloudManagerPool-714-thread-10) [    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014266 DEBUG (simCloudManagerPool-714-thread-10) [    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 1014266 DEBUG (simCloudManagerPool-714-thread-10) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014266 DEBUG (simCloudManagerPool-714-thread-10) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 1014266 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
   [junit4]   2> 1014267 INFO  (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1014267 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 1014267 DEBUG (ScheduledTrigger-715-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger2 with currently live nodes: 3
   [junit4]   2> 1014268 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1014268 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1014268 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1014268 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10004_solr
   [junit4]   2> 1014268 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1014268 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1014301 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 1014301 DEBUG (ScheduledTrigger-715-thread-2) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1014301 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1014301 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1014301 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1014301 DEBUG (ScheduledTrigger-715-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 2
   [junit4]   2> 1014301 DEBUG (ScheduledTrigger-715-thread-4) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10004_solr
   [junit4]   2> 1014301 DEBUG (ScheduledTrigger-715-thread-4) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10004_solr]
   [junit4]   2> 1014301 DEBUG (ScheduledTrigger-715-thread-4) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"19af83b854f16Tdlh6rgdzkoyndpp6uu9gvesa0",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":451865917869846,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451865917869846],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1014302 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger2 with currently live nodes: 2
   [junit4]   2> 1014302 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10004_solr
   [junit4]   2> 1014302 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10004_solr]
   [junit4]   2> 1014302 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"19af83d036e86Tdlh6rgdzkoyndpp6uu9gvesa1",
   [junit4]   2>   "source":"node_lost_trigger2",
   [junit4]   2>   "eventTime":451865942912646,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451865942912646],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1014393 DEBUG (ScheduledTrigger-715-thread-3) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger processor for lost nodes: [127.0.0.1:10004_solr] is not ready, will try later
   [junit4]   2> 1014410 DEBUG (AutoscalingActionExecutor-716-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"19af83b854f16Tdlh6rgdzkoyndpp6uu9gvesa0",
   [junit4]   2>   "source":"node_lost_trigger1",
   [junit4]   2>   "eventTime":451865917869846,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[451865917869846],
   [junit4]   2>     "_enqueue_time_":451865925492996,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1014410 INFO  (AutoscalingActionExecutor-716-thread-1) [    ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_lost_trigger1
   [junit4]   2> 1014413 DEBUG (AutoscalingActionExecutor-716-thread-1) [    ] o.a.s.c.a.ScheduledTriggers -- processing took 121 ms for event id=19af83b854f16Tdlh6rgdzkoyndpp6uu9gvesa0
   [junit4]   2> 1014430 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 1014430 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 1014430 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1014430 DEBUG (ScheduledTrigger-715-thread-1) [    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 101

[...truncated too long message...]

[junit4]   2> 5876890 INFO  (indexFetcher-10216-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 5876891 ERROR (indexFetcher-10210-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: SolrCoreState already closed.
   [junit4]   2> at org.apache.solr.update.DefaultSolrCoreState.lock(DefaultSolrCoreState.java:186)
   [junit4]   2> at org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:227)
   [junit4]   2> at org.apache.solr.update.DirectUpdateHandler2.newIndexWriter(DirectUpdateHandler2.java:766)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:646)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$12(ReplicationHandler.java:1159)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 5876891 ERROR (indexFetcher-10216-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: Index fetch failed :
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:672)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$12(ReplicationHandler.java:1159)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.openNewSearcherAndUpdateCommitPoint(IndexFetcher.java:905)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:649)
   [junit4]   2> ... 10 more
   [junit4]   2>
   [junit4]   2> 5888990 ERROR (coreCloseExecutor-10220-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Timeout waiting for all directory ref counts to be released - gave up waiting on CachedDir<<refCount=1;path=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_6188612105A09390-001/solr-instance-032/./collection1/data/index.20180115044551281;done=false>>
   [junit4]   2> 5888990 ERROR (coreCloseExecutor-10220-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Error closing directory:org.apache.solr.common.SolrException: Timeout waiting for all directory ref counts to be released - gave up waiting on CachedDir<<refCount=1;path=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_6188612105A09390-001/solr-instance-032/./collection1/data/index.20180115044551281;done=false>>
   [junit4]   2> at org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:178)
   [junit4]   2> at org.apache.solr.core.SolrCore.close(SolrCore.java:1633)
   [junit4]   2> at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:141)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 5888990 ERROR (coreCloseExecutor-10220-thread-1) [    x:collection1] o.a.s.c.SolrCore java.lang.AssertionError: 1
   [junit4]   2> at org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:192)
   [junit4]   2> at org.apache.solr.core.SolrCore.close(SolrCore.java:1633)
   [junit4]   2> at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:141)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 5888990 ERROR (coreCloseExecutor-10220-thread-1) [    x:collection1] o.a.s.c.SolrCores Error shutting down core:java.lang.AssertionError: 1
   [junit4]   2> at org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:192)
   [junit4]   2> at org.apache.solr.core.SolrCore.close(SolrCore.java:1633)
   [junit4]   2> at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:141)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 5888991 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6188612105A09390]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6c33eb26{/solr,null,UNAVAILABLE}
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_6188612105A09390-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name=PostingsFormat(name=Memory), id=PostingsFormat(name=Memory), newname=PostingsFormat(name=Memory)}, docValues:{}, maxPointsInLeafNode=1194, maxMBSortInHeap=6.955543651530648, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1370b56d), locale=es-AR, timezone=America/Argentina/Tucuman
   [junit4]   2> NOTE: Linux 4.4.0-104-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=384106616,total=529006592
   [junit4]   2> NOTE: All tests run in this JVM: [CollectionStateFormat2Test, TestUseDocValuesAsStored, CloudMLTQParserTest, TestUtils, TestPullReplica, TestSolrXml, TestDistributedSearch, BasicZkTest, TestBinaryField, TestWordDelimiterFilterFactory, TestFastOutputStream, TestCrossCoreJoin, TestFieldCacheReopen, VersionInfoTest, TestTriggerIntegration, TestComputePlanAction, TestSchemaSimilarityResource, DateMathParserTest, TestLeaderInitiatedRecoveryThread, TestSolrConfigHandlerConcurrent, DocumentBuilderTest, TestSweetSpotSimilarityFactory, TestPostingsSolrHighlighter, TestConfig, ForceLeaderTest, TermsComponentTest, TestSolr4Spatial2, CustomHighlightComponentTest, SubstringBytesRefFilterTest, ZkControllerTest, ExplicitHLLTest, NoCacheHeaderTest, TestHdfsCloudBackupRestore, RequestLoggingTest, SSLMigrationTest, JavabinLoaderTest, RAMDirectoryFactoryTest, TestMergePolicyConfig, TestSegmentSorting, RecoveryAfterSoftCommitTest, ChangedSchemaMergeTest, CursorPagingTest, TestComponentsName, TestStressLiveNodes, CoreAdminRequestStatusTest, DataDrivenBlockJoinTest, TestObjectReleaseTracker, HdfsCollectionsAPIDistributedZkTest, TestCloudPivotFacet, DirectUpdateHandlerOptimizeTest, TestInPlaceUpdatesDistrib, MultiTermTest, TestSolrCloudWithHadoopAuthPlugin, TestSolrDeletionPolicy1, NodeMutatorTest, AnalysisAfterCoreReloadTest, FacetPivotSmallTest, TestJoin, JSONWriterTest, TestManagedSchema, TestSQLHandler, TestBadConfig, TestGenericDistributedQueue, CdcrBootstrapTest, QueryParsingTest, MergeStrategyTest, TestSubQueryTransformerCrossCore, ResponseLogComponentTest, TestStandardQParsers, TestReplicationHandlerBackup, UpdateRequestProcessorFactoryTest, DeleteInactiveReplicaTest, TestStreamBody, AddBlockUpdateTest, TestUniqueKeyFieldResource, V2StandaloneTest, HdfsBasicDistributedZkTest, AnalyticsQueryTest, ClassificationUpdateProcessorTest, GraphQueryTest, TestReplicationHandler]
   [junit4] Completed [721/767 (4!)] on J0 in 336.27s, 16 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 48150 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1450 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1450/

10 tests failed.
FAILED:  org.apache.lucene.classification.CachingNaiveBayesClassifierTest.testPerformance

Error Message:
evaluation took more than 1m: 67s

Stack Trace:
java.lang.AssertionError: evaluation took more than 1m: 67s
        at __randomizedtesting.SeedInfo.seed([595E05B5607CAA2A:9EBFF7970BC89285]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.lucene.classification.CachingNaiveBayesClassifierTest.testPerformance(CachingNaiveBayesClassifierTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.lucene.index.TestIndexWriterOnVMError.testCheckpoint

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([3A0AB42C265184EF]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterOnVMError

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([3A0AB42C265184EF]:0)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testCreateDelete {seed=[5F630DCB5382D47D:4473FC87A5722C4D]}

Error Message:
Error from server at http://127.0.0.1:33503/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000036

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33503/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000036
        at __randomizedtesting.SeedInfo.seed([5F630DCB5382D47D:4473FC87A5722C4D]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.TestTlogReplica.testCreateDelete(TestTlogReplica.java:212)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testCreateDelete {seed=[5F630DCB5382D47D:F025407B91B0E761]}

Error Message:
Could not find collection : tlog_replica_test_create_delete

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : tlog_replica_test_create_delete
        at __randomizedtesting.SeedInfo.seed([5F630DCB5382D47D:F025407B91B0E761]:0)
        at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
        at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
        at org.apache.solr.cloud.TestTlogReplica.testCreateDelete(TestTlogReplica.java:188)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.RollingRestartTest.test

Error Message:
KeeperErrorCode = Session expired for /overseer_elect/leader

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
        at __randomizedtesting.SeedInfo.seed([5F630DCB5382D47D:D7373211FD7EB985]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.cloud.OverseerTaskProcessor.getLeaderId(OverseerTaskProcessor.java:379)
        at org.apache.solr.cloud.OverseerTaskProcessor.getLeaderNode(OverseerTaskProcessor.java:370)
        at org.apache.solr.cloud.RollingRestartTest.waitUntilOverseerDesignateIsLeader(RollingRestartTest.java:137)
        at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:93)
        at org.apache.solr.cloud.RollingRestartTest.test(RollingRestartTest.java:52)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([5F630DCB5382D47D:436270462627AAEE]:0)
        at org.apache.solr.cloud.TestTlogReplica.getSolrCore(TestTlogReplica.java:863)
        at org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes(TestTlogReplica.java:467)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([5F630DCB5382D47D:D9A2F5260CD3029D]:0)
        at org.apache.solr.cloud.TestTlogReplica.getSolrCore(TestTlogReplica.java:863)
        at org.apache.solr.cloud.TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates(TestTlogReplica.java:662)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:37686_solr wasn't created

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:37686_solr wasn't created
        at __randomizedtesting.SeedInfo.seed([5F630DCB5382D47D:47D985C75DB71992]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration(TriggerIntegrationTest.java:936)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testCooldown

Error Message:
[CapturedEvent{timestamp=1516122924796000000, stage=SUCCEEDED, actionName='null', event={   "id":"3a2cd50fd3abc3T6ic5944pderri0z525qegwoe5",   "source":"node_added_cooldown_trigger",   "eventTime":16374842254404547,   "eventType":"NODEADDED",   "properties":{     "eventTimes":[16374842254404547],     "_enqueue_time_":16374843273542059,     "nodeNames":["127.0.0.1:53653_solr"]}}, context={}, config={   "trigger":"node_added_cooldown_trigger",   "afterAction":[],   "stage":[     "FAILED",     "SUCCEEDED",     "IGNORED"],   "class":"org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerListener",   "beforeAction":[]}, message='null'}]

Stack Trace:
java.lang.AssertionError: [CapturedEvent{timestamp=1516122924796000000, stage=SUCCEEDED, actionName='null', event={
  "id":"3a2cd50fd3abc3T6ic5944pderri0z525qegwoe5",
  "source":"node_added_cooldown_trigger",
  "eventTime":16374842254404547,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[16374842254404547],
    "_enqueue_time_":16374843273542059,
    "nodeNames":["127.0.0.1:53653_solr"]}}, context={}, config={
  "trigger":"node_added_cooldown_trigger",
  "afterAction":[],
  "stage":[
    "FAILED",
    "SUCCEEDED",
    "IGNORED"],
  "class":"org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerListener",
  "beforeAction":[]}, message='null'}]
        at __randomizedtesting.SeedInfo.seed([5F630DCB5382D47D:6EDD602F2D28A18F]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testCooldown(TriggerIntegrationTest.java:1197)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1320 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnVMError
   [junit4]   2> ene 16, 2018 10:09:23 AM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestIndexWriterOnVMError
   [junit4]   2>    1) Thread[id=2844, name=TEST-TestIndexWriterOnVMError.testCheckpoint-seed#[3A0AB42C265184EF], state=RUNNABLE, group=TGRP-TestIndexWriterOnVMError]
   [junit4]   2>         at java.lang.Throwable.getStackTraceElement(Native Method)
   [junit4]   2>         at java.lang.Throwable.getOurStackTrace(Throwable.java:827)
   [junit4]   2>         at java.lang.Throwable.getStackTrace(Throwable.java:816)
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterOnVMError$4.eval(TestIndexWriterOnVMError.java:283)
   [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.maybeThrowDeterministicException(MockDirectoryWrapper.java:1022)
   [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:642)
   [junit4]   2>         at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]   2>         at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.<init>(DirectDocValuesConsumer.java:56)
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesFormat.fieldsConsumer(DirectDocValuesFormat.java:71)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:211)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addBinaryField(PerFieldDocValuesFormat.java:114)
   [junit4]   2>         at org.apache.lucene.index.BinaryDocValuesWriter.flush(BinaryDocValuesWriter.java:136)
   [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:247)
   [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:131)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:557)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:673)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453)
   [junit4]   2>         at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterOnVMError.doTest(TestIndexWriterOnVMError.java:175)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterOnVMError.testCheckpoint(TestIndexWriterOnVMError.java:280)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>    2) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:636)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:493)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:251)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:368)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    3) Thread[id=2843, name=SUITE-TestIndexWriterOnVMError-seed#[3A0AB42C265184EF], state=RUNNABLE, group=TGRP-TestIndexWriterOnVMError]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1559)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:705)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:139)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:626)
   [junit4]   2>    4) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterOnVMError -Dtests.method=testCheckpoint -Dtests.seed=3A0AB42C265184EF -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-EC -Dtests.timezone=Europe/Ljubljana -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   7141s J1 | TestIndexWriterOnVMError.testCheckpoint <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > at __randomizedtesting.SeedInfo.seed([3A0AB42C265184EF]:0)
   [junit4]   2> ene 16, 2018 10:09:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> ene 16, 2018 10:09:44 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.index.TestIndexWriterOnVMError:
   [junit4]   2>    1) Thread[id=2844, name=TEST-TestIndexWriterOnVMError.testCheckpoint-seed#[3A0AB42C265184EF], state=RUNNABLE, group=TGRP-TestIndexWriterOnVMError]
   [junit4]   2>         at java.lang.Thread.yield(Native Method)
   [junit4]   2>         at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:138)
   [junit4]   2>         at org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:126)
   [junit4]   2>         at org.apache.lucene.store.DataOutput.writeSignedVLong(DataOutput.java:236)
   [junit4]   2>         at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:227)
   [junit4]   2>         at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addTermsDict(Lucene70DocValuesConsumer.java:442)
   [junit4]   2>         at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.doAddSortedField(Lucene70DocValuesConsumer.java:437)
   [junit4]   2>         at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedField(Lucene70DocValuesConsumer.java:394)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedField(PerFieldDocValuesFormat.java:119)
   [junit4]   2>         at org.apache.lucene.index.SortedDocValuesWriter.flush(SortedDocValuesWriter.java:163)
   [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:247)
   [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:131)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:557)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:673)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453)
   [junit4]   2>         at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterOnVMError.doTest(TestIndexWriterOnVMError.java:175)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterOnVMError.testCheckpoint(TestIndexWriterOnVMError.java:280)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> ene 16, 2018 10:09:44 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=2844, name=TEST-TestIndexWriterOnVMError.testCheckpoint-seed#[3A0AB42C265184EF], state=RUNNABLE, group=TGRP-TestIndexWriterOnVMError]
   [junit4]   2> ene 16, 2018 10:09:47 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=2844, name=TEST-TestIndexWriterOnVMError.testCheckpoint-seed#[3A0AB42C265184EF], state=RUNNABLE, group=TGRP-TestIndexWriterOnVMError]
   [junit4]   2>         at org.apache.lucene.store.MockIndexInputWrapper.getFilePointer(MockIndexInputWrapper.java:122)
   [junit4]   2>         at org.apache.lucene.store.BufferedChecksumIndexInput.getFilePointer(BufferedChecksumIndexInput.java:65)
   [junit4]   2>         at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:521)
   [junit4]   2>         at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.checkIntegrity(Lucene50PostingsReader.java:1286)
   [junit4]   2>         at org.apache.lucene.codecs.blockterms.BlockTermsReader.checkIntegrity(BlockTermsReader.java:870)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:348)
   [junit4]   2>         at org.apache.lucene.index.CodecReader.checkIntegrity(CodecReader.java:286)
   [junit4]   2>         at org.apache.lucene.util.TestUtil.checkReader(TestUtil.java:330)
   [junit4]   2>         at org.apache.lucene.util.TestUtil.checkReader(TestUtil.java:319)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterOnVMError.doTest(TestIndexWriterOnVMError.java:176)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterOnVMError.testCheckpoint(TestIndexWriterOnVMError.java:280)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {text_payloads=PostingsFormat(name=LuceneFixedGap), text_vectors=PostingsFormat(name=LuceneVarGapFixedInterval), text1=PostingsFormat(name=LuceneVarGapDocFreqInterval), id=BlockTreeOrds(blocksize=128)}, docValues:{dv3=DocValuesFormat(name=Lucene70), dv2=DocValuesFormat(name=Direct), dv5=DocValuesFormat(name=Asserting), dv=DocValuesFormat(name=Memory), text_payloads=DocValuesFormat(name=Lucene70), dv4=DocValuesFormat(name=Memory), text_vectors=DocValuesFormat(name=Memory), text1=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Asserting), point2d=DocValuesFormat(name=Memory), stored1=DocValuesFormat(name=Memory), point=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=467, maxMBSortInHeap=7.849047930627821, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@428e7648), locale=es-EC, timezone=Europe/Ljubljana
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=2,free=322709376,total=498073600
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndependenceChiSquared, TestGraphTokenizers, TestAddIndexes, TestSpanBoostQuery, TestDoc, TestStressDeletes, TestComplexExplanations, TestMultiCollector, TestOrdinalMap, TestIndexingSequenceNumbers, TestLucene50StoredFieldsFormat, TestDoubleRangeFieldQueries, TestIndexWriterLockRelease, TestFastDecompressionMode, TestFilterDirectory, TestIndexWriterOnVMError]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterOnVMError -Dtests.seed=3A0AB42C265184EF -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-EC -Dtests.timezone=Europe/Ljubljana -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestIndexWriterOnVMError (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > at __randomizedtesting.SeedInfo.seed([3A0AB42C265184EF]:0)
   [junit4] Completed [238/476 (1!)] on J1 in 7223.75s, 2 tests, 2 errors <<< FAILURES!

[...truncated 5874 lines...]
   [junit4] Suite: org.apache.lucene.classification.CachingNaiveBayesClassifierTest
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CachingNaiveBayesClassifierTest -Dtests.method=testPerformance -Dtests.seed=595E05B5607CAA2A -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=sr-BA -Dtests.timezone=America/Knox_IN -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 85.6s J2 | CachingNaiveBayesClassifierTest.testPerformance <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: evaluation took more than 1m: 67s
   [junit4]    > at __randomizedtesting.SeedInfo.seed([595E05B5607CAA2A:9EBFF7970BC89285]:0)
   [junit4]    > at org.apache.lucene.classification.CachingNaiveBayesClassifierTest.testPerformance(CachingNaiveBayesClassifierTest.java:111)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {bool=PostingsFormat(name=Memory), cat=PostingsFormat(name=Memory), text=PostingsFormat(name=Memory)}, docValues:{}, maxPointsInLeafNode=1696, maxMBSortInHeap=7.245235214871238, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1c4eb0cf), locale=sr-BA, timezone=America/Knox_IN
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=187893808,total=340787200
   [junit4]   2> NOTE: All tests run in this JVM: [CachingNaiveBayesClassifierTest]
   [junit4] Completed [3/11 (1!)] on J2 in 98.16s, 4 tests, 1 failure <<< FAILURES!

[...truncated 5760 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
   [junit4]   2> 514599 INFO  (SUITE-TriggerIntegrationTest-seed#[5F630DCB5382D47D]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5F630DCB5382D47D-001/init-core-data-001
   [junit4]   2> 514600 WARN  (SUITE-TriggerIntegrationTest-seed#[5F630DCB5382D47D]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 514600 INFO  (SUITE-TriggerIntegrationTest-seed#[5F630DCB5382D47D]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 514638 INFO  (SUITE-TriggerIntegrationTest-seed#[5F630DCB5382D47D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 514641 INFO  (SUITE-TriggerIntegrationTest-seed#[5F630DCB5382D47D]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5F630DCB5382D47D-001/tempDir-001
   [junit4]   2> 514682 INFO  (SUITE-TriggerIntegrationTest-seed#[5F630DCB5382D47D]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 514683 INFO  (Thread-188) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 514683 INFO  (Thread-188) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 514783 INFO  (SUITE-TriggerIntegrationTest-seed#[5F630DCB5382D47D]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:39204
   [junit4]   2> 514811 ERROR (Thread-188) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 515726 INFO  (zkConnectionManagerCallback-222-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 515772 INFO  (jetty-launcher-219-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 515783 INFO  (jetty-launcher-219-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7cbde106{/solr,null,AVAILABLE}
   [junit4]   2> 515885 INFO  (jetty-launcher-219-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@16e4b5ee{SSL,[ssl, http/1.1]}{127.0.0.1:37572}
   [junit4]   2> 515885 INFO  (jetty-launcher-219-thread-2) [    ] o.e.j.s.Server Started @544194ms
   [junit4]   2> 515885 INFO  (jetty-launcher-219-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37572}
   [junit4]   2> 515886 ERROR (jetty-launcher-219-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 515886 INFO  (jetty-launcher-219-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 515886 INFO  (jetty-launcher-219-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 515886 INFO  (jetty-launcher-219-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 515887 INFO  (jetty-launcher-219-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-16T17:11:55.184Z
   [junit4]   2> 515908 INFO  (jetty-launcher-219-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 516834 INFO  (jetty-launcher-219-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@39295559{/solr,null,AVAILABLE}
   [junit4]   2> 516835 INFO  (jetty-launcher-219-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4dfb8207{SSL,[ssl, http/1.1]}{127.0.0.1:54434}
   [junit4]   2> 517119 INFO  (jetty-launcher-219-thread-1) [    ] o.e.j.s.Server Started @545427ms
   [junit4]   2> 517119 INFO  (jetty-launcher-219-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54434}
   [junit4]   2> 517119 ERROR (jetty-launcher-219-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 517119 INFO  (jetty-launcher-219-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 517120 INFO  (jetty-launcher-219-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 517120 INFO  (jetty-launcher-219-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 517120 INFO  (jetty-launcher-219-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-16T17:11:56.418Z
   [junit4]   2> 517135 INFO  (zkConnectionManagerCallback-224-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 517138 INFO  (jetty-launcher-219-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 517352 INFO  (zkConnectionManagerCallback-226-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 517397 INFO  (jetty-launcher-219-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 517594 INFO  (jetty-launcher-219-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39204/solr
   [junit4]   2> 517626 INFO  (jetty-launcher-219-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39204/solr
   [junit4]   2> 517813 INFO  (zkConnectionManagerCallback-234-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 517883 INFO  (zkConnectionManagerCallback-232-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 517996 INFO  (zkConnectionManagerCallback-236-thread-1-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 518050 INFO  (zkConnectionManagerCallback-238-thread-1-processing-n:127.0.0.1:54434_solr) [n:127.0.0.1:54434_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 518236 INFO  (jetty-launcher-219-thread-2) [n:127.0.0.1:37572_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 518237 INFO  (jetty-launcher-219-thread-2) [n:127.0.0.1:37572_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37572_solr
   [junit4]   2> 518239 INFO  (jetty-launcher-219-thread-2) [n:127.0.0.1:37572_solr    ] o.a.s.c.Overseer Overseer (id=99360618185359365-127.0.0.1:37572_solr-n_0000000000) starting
   [junit4]   2> 518430 INFO  (jetty-launcher-219-thread-2) [n:127.0.0.1:37572_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37572_solr
   [junit4]   2> 518430 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 518511 INFO  (OverseerStateUpdate-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 518530 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 518542 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:37572_solr]
   [junit4]   2> 518542 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 518542 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 518790 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 518798 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 518814 DEBUG (ScheduledTrigger-362-thread-1) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
   [junit4]   2> 519641 INFO  (jetty-launcher-219-thread-1) [n:127.0.0.1:54434_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 519663 INFO  (jetty-launcher-219-thread-1) [n:127.0.0.1:54434_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 519679 INFO  (jetty-launcher-219-thread-1) [n:127.0.0.1:54434_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54434_solr
   [junit4]   2> 519862 INFO  (zkCallback-237-thread-1-processing-n:127.0.0.1:54434_solr) [n:127.0.0.1:54434_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 519862 INFO  (zkCallback-235-thread-2-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 519970 DEBUG (ScheduledTrigger-362-thread-1) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 520160 INFO  (jetty-launcher-219-thread-2) [n:127.0.0.1:37572_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37572.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@b5a57b3
   [junit4]   2> 520288 INFO  (jetty-launcher-219-thread-2) [n:127.0.0.1:37572_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37572.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@b5a57b3
   [junit4]   2> 520288 INFO  (jetty-launcher-219-thread-2) [n:127.0.0.1:37572_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37572.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@b5a57b3
   [junit4]   2> 520326 INFO  (jetty-launcher-219-thread-2) [n:127.0.0.1:37572_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5F630DCB5382D47D-001/tempDir-001/node2/.
   [junit4]   2> 520979 DEBUG (ScheduledTrigger-362-thread-2) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 521121 INFO  (jetty-launcher-219-thread-1) [n:127.0.0.1:54434_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54434.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@b5a57b3
   [junit4]   2> 521198 INFO  (jetty-launcher-219-thread-1) [n:127.0.0.1:54434_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54434.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@b5a57b3
   [junit4]   2> 521198 INFO  (jetty-launcher-219-thread-1) [n:127.0.0.1:54434_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54434.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@b5a57b3
   [junit4]   2> 521200 INFO  (jetty-launcher-219-thread-1) [n:127.0.0.1:54434_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_5F630DCB5382D47D-001/tempDir-001/node1/.
   [junit4]   2> 521618 INFO  (zkConnectionManagerCallback-244-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 521638 INFO  (zkConnectionManagerCallback-248-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 521640 INFO  (SUITE-TriggerIntegrationTest-seed#[5F630DCB5382D47D]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 521641 INFO  (SUITE-TriggerIntegrationTest-seed#[5F630DCB5382D47D]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39204/solr ready
   [junit4]   2> 521751 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerThrottling
   [junit4]   2> 521770 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
   [junit4]   2> 521782 DEBUG (zkCallback-235-thread-2-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 521782 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 521806 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 521808 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 521809 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 526321 INFO  (qtp751715126-963) [n:127.0.0.1:37572_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=92
   [junit4]   2> 526417 INFO  (qtp1835172169-974) [n:127.0.0.1:54434_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 526419 DEBUG (qtp1835172169-971) [n:127.0.0.1:54434_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 526436 DEBUG (zkCallback-235-thread-2-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 526470 DEBUG (zkCallback-235-thread-2-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:54434_solr, 127.0.0.1:37572_solr]
   [junit4]   2> 526470 DEBUG (zkCallback-235-thread-2-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 526471 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 526473 INFO  (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 526474 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 526475 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 526475 DEBUG (ScheduledTrigger-362-thread-3) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 526475 DEBUG (ScheduledTrigger-362-thread-3) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 526794 INFO  (qtp1835172169-971) [n:127.0.0.1:54434_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=4586
   [junit4]   2> 526886 INFO  (qtp751715126-956) [n:127.0.0.1:37572_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 527073 INFO  (qtp1835172169-967) [n:127.0.0.1:54434_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=177
   [junit4]   2> 527370 DEBUG (qtp1835172169-971) [n:127.0.0.1:54434_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 527374 INFO  (qtp1835172169-971) [n:127.0.0.1:54434_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=572
   [junit4]   2> 527376 DEBUG (zkCallback-235-thread-2-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 527406 DEBUG (zkCallback-235-thread-2-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:54434_solr, 127.0.0.1:37572_solr]
   [junit4]   2> 527406 DEBUG (zkCallback-235-thread-2-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 527407 DEBUG (zkCallback-235-thread-2-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:54434_solr, 127.0.0.1:37572_solr]
   [junit4]   2> 527407 DEBUG (zkCallback-235-thread-2-processing-n:127.0.0.1:37572_solr) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
   [junit4]   2> 527407 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 527409 INFO  (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 527419 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 527421 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@cf6a9c{/solr,null,AVAILABLE}
   [junit4]   2> 527427 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 527427 DEBUG (ScheduledTrigger-362-thread-4) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
   [junit4]   2> 527427 DEBUG (ScheduledTrigger-362-thread-4) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 527427 DEBUG (OverseerAutoScalingTriggerThread-99360618185359365-127.0.0.1:37572_solr-n_0000000000) [n:127.0.0.1:37572_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 527428 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@317f2917{SSL,[ssl, http/1.1]}{127.0.0.1:51324}
   [junit4]   2> 527428 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.e.j.s.Server Started @555737ms
   [junit4]   2> 527428 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51324}
   [junit4]   2> 527429 ERROR (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 527429 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 527429 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 527429 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 527429 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-01-16T17:12:06.727Z
   [junit4]   2> 527453 INFO  (zkConnectionManagerCallback-250-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 527463 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 527513 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39204/solr
   [junit4]   2> 527571 INFO  (zkConnectionManagerCallback-254-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 527626 INFO  (zkConnectionManagerCallback-256-thread-1-processing-n:127.0.0.1:51324_solr) [n:127.0.0.1:51324_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 527668 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [n:127.0.0.1:51324_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 527688 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D]) [n:127.0.0.1:51324_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 527694 INFO  (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[5F630DCB5382D47D

[...truncated too long message...]

/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/bcprov-jdk15on-1.54.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/byte-buddy-1.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-math3-3.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mockito-core-2.6.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/objenesis-2.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-8.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/backward-codecs/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/contrib/analysis-extras/lib/icu4j-60.2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_144/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.3.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20180116_170250_4008500111829940324945.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20180116_170250_4006102885239678798602.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Forked process returned with error code: 1. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20180116_170250_400520242532759884893.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20180116_170250_400635540163281974962.syserr
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1519)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:844: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:780: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build.xml:262: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/common-build.xml:556: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1513: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1038: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 1354 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-NightlyTests-master #1449 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]