[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 7 - Failure

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 7 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/7/

22 tests failed.
FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test15Seconds

Error Message:
Core 00005_core bad! expected:<1328> but was:<906>

Stack Trace:
java.lang.AssertionError: Core 00005_core bad! expected:<1328> but was:<906>
        at __randomizedtesting.SeedInfo.seed([CD9376F4F7291608:37A926ACE74455F4]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.apache.solr.core.OpenCloseCoreStressTest.checkResults(OpenCloseCoreStressTest.java:280)
        at org.apache.solr.core.OpenCloseCoreStressTest.doStress(OpenCloseCoreStressTest.java:179)
        at org.apache.solr.core.OpenCloseCoreStressTest.test15Seconds(OpenCloseCoreStressTest.java:117)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test10Minutes

Error Message:
Core 00005_core bad! expected:<3651> but was:<729>

Stack Trace:
java.lang.AssertionError: Core 00005_core bad! expected:<3651> but was:<729>
        at __randomizedtesting.SeedInfo.seed([CD9376F4F7291608:DD16A25D658CDC23]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.apache.solr.core.OpenCloseCoreStressTest.checkResults(OpenCloseCoreStressTest.java:280)
        at org.apache.solr.core.OpenCloseCoreStressTest.doStress(OpenCloseCoreStressTest.java:179)
        at org.apache.solr.core.OpenCloseCoreStressTest.test10Minutes(OpenCloseCoreStressTest.java:123)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
        at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
        at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
        at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
        at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
        at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.finishTerm(Lucene50PostingsWriter.java:388)
        at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:169)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
        at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
        at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
        at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:413)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
        at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:189)
        at org.apache.solr.uninverting.TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit(TestDocTermOrdsUninvertLimit.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)


FAILED:  junit.framework.TestSuite.org.apache.solr.uninverting.TestLegacyFieldCache

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.reflect.Method.copy(Method.java:153)
        at java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:140)
        at sun.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:316)
        at java.lang.Class.copyMethods(Class.java:3124)
        at java.lang.Class.getDeclaredMethods(Class.java:1975)
        at org.junit.internal.MethodSorter.getDeclaredMethods(MethodSorter.java:54)
        at org.junit.runners.model.TestClass.scanAnnotatedMembers(TestClass.java:65)
        at org.junit.runners.model.TestClass.<init>(TestClass.java:57)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$12.run(RandomizedRunner.java:1084)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$12.run(RandomizedRunner.java:1081)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.getAnnotatedFieldValues(RandomizedRunner.java:1081)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.withClassRules(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:704)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:629)


FAILED:  org.apache.solr.uninverting.TestNumericTerms32.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.uninverting.TestNumericTerms64.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.uninverting.TestUninvertingReader.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.DataDrivenBlockJoinTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.MaxSizeAutoCommitTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.SoftAutoCommitTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.SolrCmdDistributorTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.TestHdfsUpdateLog.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.VersionInfoTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.processor.ClassificationUpdateProcessorFactoryTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.processor.ClassificationUpdateProcessorIntegrationTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.processor.FieldMutatingUpdateProcessorTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.processor.RegexBoostProcessorTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.processor.TestDocBasedVersionConstraints.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.update.processor.URLClassifyProcessorTest.initializationError

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  junit.framework.TestSuite.org.apache.solr.util.PrimUtilsTest

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  junit.framework.TestSuite.org.apache.solr.util.hll.HLLSerializationTest

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded




Build Log:
[...truncated 15245 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4]   2> 3569739 INFO  (SUITE-OpenCloseCoreStressTest-seed#[CD9376F4F7291608]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/init-core-data-001
   [junit4]   2> 3569740 INFO  (SUITE-OpenCloseCoreStressTest-seed#[CD9376F4F7291608]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 3569741 INFO  (SUITE-OpenCloseCoreStressTest-seed#[CD9376F4F7291608]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4] IGNOR/A 0.00s J1 | OpenCloseCoreStressTest.test1Hour
   [junit4]    > Assumption #1: 'weekly' test group is disabled (@Weekly())
   [junit4]   2> 3569753 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test15Seconds
   [junit4]   2> 3569754 WARN  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 3569771 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 3569771 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 3569771 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 3569772 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3569772 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3569772 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 3569773 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@39f0d30{/solr,null,AVAILABLE}
   [junit4]   2> 3569780 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@acc7ea8{HTTP/1.1,[http/1.1]}{127.0.0.1:43117}
   [junit4]   2> 3569780 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.e.j.s.Server Started @3569844ms
   [junit4]   2> 3569780 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43117}
   [junit4]   2> 3569780 ERROR (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3569780 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3569781 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.0
   [junit4]   2> 3569781 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 3569781 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3569781 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-09T00:37:15.189Z
   [junit4]   2> 3569781 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/solr.xml
   [junit4]   2> 3569786 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3569961 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 3569995 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3569995 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3570024 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d
   [junit4]   2> 3570043 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d
   [junit4]   2> 3570043 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d
   [junit4]   2> 3570052 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.c.CorePropertiesLocator Found 7 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001
   [junit4]   2> 3570052 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [00000_core, 00004_core, 00003_core, 00002_core, 00006_core, 00001_core, 00005_core]
   [junit4]   2> 3570093 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.c.OpenCloseCoreStressTest Starting indexing and querying
   [junit4]   2> 3570093 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[CD9376F4F7291608]) [    ] o.a.s.c.OpenCloseCoreStressTest
   [junit4]   2>
   [junit4]   2>
   [junit4]   2>
   [junit4]   2>
   [junit4]   2> Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 3570098 INFO  (Thread-93207) [    ] o.a.s.c.OneIndexer Starting indexing thread: 129425
   [junit4]   2> 3570098 INFO  (Thread-93208) [    ] o.a.s.c.OneIndexer Starting indexing thread: 129426
   [junit4]   2> 3570101 INFO  (Thread-93209) [    ] o.a.s.c.OneIndexer Starting indexing thread: 129427
   [junit4]   2> 3570101 INFO  (Thread-93211) [    ] o.a.s.c.OneIndexer Starting indexing thread: 129429
   [junit4]   2> 3570102 INFO  (Thread-93210) [    ] o.a.s.c.OneIndexer Starting indexing thread: 129428
   [junit4]   2> 3570102 INFO  (Thread-93212) [    ] o.a.s.c.OneIndexer Starting indexing thread: 129430
   [junit4]   2> 3570102 INFO  (Thread-93213) [    ] o.a.s.c.OneIndexer Starting indexing thread: 129432
   [junit4]   2> 3570103 INFO  (Thread-93214) [    ] o.a.s.c.OneIndexer Starting indexing thread: 129433
   [junit4]   2> 3570106 INFO  (Thread-93215) [    ] o.a.s.c.OneIndexer Starting indexing thread: 129434
   [junit4]   2> 3570110 INFO  (Thread-93218) [    ] o.a.s.c.OneQuery Starting query thread: 129437
   [junit4]   2> 3570110 INFO  (Thread-93219) [    ] o.a.s.c.OneQuery Starting query thread: 129439
   [junit4]   2> 3570111 INFO  (Thread-93222) [    ] o.a.s.c.OneQuery Starting query thread: 129440
   [junit4]   2> 3570114 INFO  (Thread-93223) [    ] o.a.s.c.OneQuery Starting query thread: 129441
   [junit4]   2> 3570114 INFO  (Thread-93224) [    ] o.a.s.c.OneQuery Starting query thread: 129442
   [junit4]   2> 3570115 INFO  (Thread-93225) [    ] o.a.s.c.OneQuery Starting query thread: 129443
   [junit4]   2> 3570115 INFO  (Thread-93226) [    ] o.a.s.c.OneQuery Starting query thread: 129444
   [junit4]   2> 3570115 INFO  (Thread-93227) [    ] o.a.s.c.OneQuery Starting query thread: 129445
   [junit4]   2> 3570116 INFO  (Thread-93228) [    ] o.a.s.c.OneQuery Starting query thread: 129446
   [junit4]   2> 3570121 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 3570121 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 3570135 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1316187103
   [junit4]   2> 3570135 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 3570135 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1706191214
   [junit4]   2> 3570140 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 3570151 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.s.IndexSchema [00006_core] Schema name=tiny
   [junit4]   2> 3570151 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.s.IndexSchema [00002_core] Schema name=tiny
   [junit4]   2> 3570152 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.s.IndexSchema [00003_core] Schema name=tiny
   [junit4]   2> 3570154 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.s.IndexSchema [00000_core] Schema name=tiny
   [junit4]   2> 3570157 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3570157 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3570157 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.c.CoreContainer Creating SolrCore '00002_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00002_core, trusted=true
   [junit4]   2> 3570157 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.c.CoreContainer Creating SolrCore '00006_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00006_core, trusted=true
   [junit4]   2> 3570157 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3570157 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3570157 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.c.CoreContainer Creating SolrCore '00003_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00003_core, trusted=true
   [junit4]   2> 3570157 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.c.CoreContainer Creating SolrCore '00000_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00000_core, trusted=true
   [junit4]   2> 3570158 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.00002_core' (registry 'solr.core.00002_core') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d
   [junit4]   2> 3570158 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.00006_core' (registry 'solr.core.00006_core') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d
   [junit4]   2> 3570158 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.00003_core' (registry 'solr.core.00003_core') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d
   [junit4]   2> 3570158 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.00000_core' (registry 'solr.core.00000_core') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d
   [junit4]   2> 3570158 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.c.SolrCore [[00006_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00006_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00006_core/data/]
   [junit4]   2> 3570158 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.c.SolrCore [[00002_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00002_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00002_core/data/]
   [junit4]   2> 3570158 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.c.SolrCore [[00003_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00003_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00003_core/data/]
   [junit4]   2> 3570158 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.c.SolrCore [[00000_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00000_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00000_core/data/]
   [junit4]   2> 3570163 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@50f6e53d
   [junit4]   2> 3570163 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@417fb7b8
   [junit4]   2> 3570165 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@16aeac6b
   [junit4]   2> 3570170 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7d1efa4f
   [junit4]   2> 3570306 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3570306 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3570308 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.1240234375, noCFSRatio=0.6879961872112529]
   [junit4]   2> 3570310 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@5d8cfa27[00000_core] main]
   [junit4]   2> 3570310 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00000_core/conf
   [junit4]   2> 3570311 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3570317 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id1]} 0 212
   [junit4]   2> 3570320 INFO  (qtp1651730374-129447) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id9]} 0 203
   [junit4]   2> 3570321 INFO  (qtp1651730374-129436) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id8]} 0 211
   [junit4]   2> 3570322 INFO  (qtp1651730374-129438) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id7]} 0 205
   [junit4]   2> 3570322 INFO  (qtp1651730374-129436) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id13]} 0 0
   [junit4]   2> 3570323 INFO  (qtp1651730374-129438) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id14]} 0 0
   [junit4]   2> 3570328 INFO  (searcherExecutor-11756-thread-1-processing-x:00000_core) [    x:00000_core] o.a.s.c.SolrCore [00000_core] Registered new searcher Searcher@5d8cfa27[00000_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3570328 INFO  (qtp1651730374-129455) [    x:00000_core] o.a.s.c.S.Request [00000_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=109
   [junit4]   2> 3570328 INFO  (qtp1651730374-129456) [    x:00000_core] o.a.s.c.S.Request [00000_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=109
   [junit4]   2> 3570335 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 3570335 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 3570342 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3570342 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3570344 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.1240234375, noCFSRatio=0.6879961872112529]
   [junit4]   2> 3570345 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@623592ac[00002_core] main]
   [junit4]   2> 3570346 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00002_core/conf
   [junit4]   2> 3570347 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3570347 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3570347 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3570349 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.1240234375, noCFSRatio=0.6879961872112529]
   [junit4]   2> 3570349 INFO  (searcherExecutor-11753-thread-1-processing-x:00002_core) [    x:00002_core] o.a.s.c.SolrCore [00002_core] Registered new searcher Searcher@623592ac[00002_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3570350 INFO  (qtp1651730374-129448) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=137
   [junit4]   2> 3570350 INFO  (qtp1651730374-129457) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=131
   [junit4]   2> 3570350 INFO  (qtp1651730374-129447) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id12]} 0 28
   [junit4]   2> 3570351 INFO  (qtp1651730374-129394) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id3]} 0 247
   [junit4]   2> 3570356 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@620617a7[00006_core] main]
   [junit4]   2> 3570356 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3570356 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3570356 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00006_core/conf
   [junit4]   2> 3570357 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3570358 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.1240234375, noCFSRatio=0.6879961872112529]
   [junit4]   2> 3570359 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@332fbee0[00003_core] main]
   [junit4]   2> 3570359 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00003_core/conf
   [junit4]   2> 3570360 INFO  (searcherExecutor-11754-thread-1-processing-x:00006_core) [    x:00006_core] o.a.s.c.SolrCore [00006_core] Registered new searcher Searcher@620617a7[00006_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3570360 INFO  (qtp1651730374-129458) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=140
   [junit4]   2> 3570360 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3570360 INFO  (qtp1651730374-129460) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=140
   [junit4]   2> 3570361 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.s.IndexSchema [00001_core] Schema name=tiny
   [junit4]   2> 3570364 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3570364 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.c.CoreContainer Creating SolrCore '00001_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00001_core, trusted=true
   [junit4]   2> 3570365 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.00001_core' (registry 'solr.core.00001_core') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d
   [junit4]   2> 3570365 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.c.SolrCore [[00001_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00001_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00001_core/data/]
   [junit4]   2> 3570370 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@4a2cd420
   [junit4]   2> 3570374 INFO  (searcherExecutor-11755-thread-1-processing-x:00003_core) [    x:00003_core] o.a.s.c.SolrCore [00003_core] Registered new searcher Searcher@332fbee0[00003_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3570375 INFO  (qtp1651730374-129453) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=161
   [junit4]   2> 3570375 INFO  (qtp1651730374-129454) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=161
   [junit4]   2> 3570375 INFO  (qtp1651730374-129459) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=155
   [junit4]   2> 3570376 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.s.IndexSchema [00005_core] Schema name=tiny
   [junit4]   2> 3570376 INFO  (qtp1651730374-129438) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id16]} 0 20
   [junit4]   2> 3570376 INFO  (qtp1651730374-129393) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id6]} 0 270
   [junit4]   2> 3570376 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id4]} 0 273
   [junit4]   2> 3570379 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3570379 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.c.CoreContainer Creating SolrCore '00005_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00005_core, trusted=true
   [junit4]   2> 3570379 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.00005_core' (registry 'solr.core.00005_core') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d
   [junit4]   2> 3570380 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.c.SolrCore [[00005_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00005_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00005_core/data/]
   [junit4]   2> 3570384 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7af7049d
   [junit4]   2> 3570394 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.0
   [junit4]   2> 3570402 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id2]} 0 298
   [junit4]   2> 3570402 INFO  (qtp1651730374-129435) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id5]} 0 291
   [junit4]   2> 3570402 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id19]} 0 0
   [junit4]   2> 3570403 INFO  (qtp1651730374-129397) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id21]} 0 0
   [junit4]   2> 3570403 INFO  (qtp1651730374-129397) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id22]} 0 0
   [junit4]   2> 3570403 INFO  (qtp1651730374-129397) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id23]} 0 0
   [junit4]   2> 3570414 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.s.IndexSchema [00004_core] Schema name=tiny
   [junit4]   2> 3570434 INFO  (qtp1651730374-129460) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570434 INFO  (qtp1651730374-129458) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570434 INFO  (qtp1651730374-129394) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id26]} 0 0
   [junit4]   2> 3570434 INFO  (qtp1651730374-129447) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id25]} 0 0
   [junit4]   2> 3570432 INFO  (qtp1651730374-129448) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id20]} 0 0
   [junit4]   2> 3570443 INFO  (qtp1651730374-129393) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id28]} 0 0
   [junit4]   2> 3570443 INFO  (qtp1651730374-129396) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id29]} 0 0
   [junit4]   2> 3570444 INFO  (qtp1651730374-129435) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id30]} 0 0
   [junit4]   2> 3570444 INFO  (qtp1651730374-129397) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id31]} 0 0
   [junit4]   2> 3570451 INFO  (qtp1651730374-129438) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570451 INFO  (qtp1651730374-129438) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570452 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3570452 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.c.CoreContainer Creating SolrCore '00004_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00004_core, trusted=true
   [junit4]   2> 3570452 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.00004_core' (registry 'solr.core.00004_core') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f8acc2d
   [junit4]   2> 3570452 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.c.SolrCore [[00004_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00004_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00004_core/data/]
   [junit4]   2> 3570469 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@122b4591
   [junit4]   2> 3570476 INFO  (qtp1651730374-129447) [    x:00000_core] o.a.s.c.S.Request [00000_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570476 INFO  (qtp1651730374-129448) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570476 INFO  (qtp1651730374-129448) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570543 INFO  (qtp1651730374-129435) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570552 INFO  (qtp1651730374-129397) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570552 INFO  (qtp1651730374-129438) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570566 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3570566 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3570568 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.1240234375, noCFSRatio=0.6879961872112529]
   [junit4]   2> 3570574 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@21afe6b9[00001_core] main]
   [junit4]   2> 3570575 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00001_core/conf
   [junit4]   2> 3570576 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3570577 INFO  (searcherExecutor-11769-thread-1-processing-x:00001_core) [    x:00001_core] o.a.s.c.SolrCore [00001_core] Registered new searcher Searcher@21afe6b9[00001_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3570577 INFO  (qtp1651730374-129396) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=35
   [junit4]   2> 3570578 INFO  (qtp1651730374-129394) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=116
   [junit4]   2> 3570579 INFO  (qtp1651730374-129460) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570579 INFO  (qtp1651730374-129460) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570579 INFO  (qtp1651730374-129460) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570592 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id15]} 0 269
   [junit4]   2> 3570592 INFO  (qtp1651730374-129453) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id32]} 0 148
   [junit4]   2> 3570592 INFO  (qtp1651730374-129455) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id17]} 0 236
   [junit4]   2> 3570593 INFO  (qtp1651730374-129393) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id34]} 0 0
   [junit4]   2> 3570593 INFO  (qtp1651730374-129397) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id36]} 0 0
   [junit4]   2> 3570594 INFO  (qtp1651730374-129438) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id37]} 0 0
   [junit4]   2> 3570594 INFO  (qtp1651730374-129396) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id38]} 0 0
   [junit4]   2> 3570614 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3570614 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3570616 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.1240234375, noCFSRatio=0.6879961872112529]
   [junit4]   2> 3570617 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@ad9bde1[00005_core] main]
   [junit4]   2> 3570618 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00005_core/conf
   [junit4]   2> 3570619 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3570634 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id10]} 0 316
   [junit4]   2> 3570634 INFO  (qtp1651730374-129447) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id41]} 0 0
   [junit4]   2> 3570635 INFO  (qtp1651730374-129448) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id42]} 0 0
   [junit4]   2> 3570636 INFO  (qtp1651730374-129453) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id43]} 0 0
   [junit4]   2> 3570636 INFO  (qtp1651730374-129436) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id44]} 0 0
   [junit4]   2> 3570637 INFO  (qtp1651730374-129455) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id45]} 0 0
   [junit4]   2> 3570637 INFO  (qtp1651730374-129393) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id46]} 0 0
   [junit4]   2> 3570638 INFO  (qtp1651730374-129397) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id47]} 0 0
   [junit4]   2> 3570638 INFO  (qtp1651730374-129438) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id48]} 0 0
   [junit4]   2> 3570639 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id49]} 0 0
   [junit4]   2> 3570652 INFO  (searcherExecutor-11770-thread-1-processing-x:00005_core) [    x:00005_core] o.a.s.c.SolrCore [00005_core] Registered new searcher Searcher@ad9bde1[00005_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3570652 INFO  (qtp1651730374-129394) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id39]} 0 57
   [junit4]   2> 3570652 INFO  (qtp1651730374-129435) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id35]} 0 59
   [junit4]   2> 3570653 INFO  (qtp1651730374-129453) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id51]} 0 0
   [junit4]   2> 3570653 INFO  (qtp1651730374-129459) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id33]} 0 209
   [junit4]   2> 3570654 INFO  (qtp1651730374-129454) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id27]} 0 209
   [junit4]   2> 3570654 INFO  (qtp1651730374-129395) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id11]} 0 334
   [junit4]   2> 3570655 INFO  (qtp1651730374-129395) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id52]} 0 0
   [junit4]   2> 3570655 INFO  (qtp1651730374-129395) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id54]} 0 0
   [junit4]   2> 3570656 INFO  (qtp1651730374-129396) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id56]} 0 0
   [junit4]   2> 3570656 INFO  (qtp1651730374-129396) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id57]} 0 0
   [junit4]   2> 3570657 INFO  (qtp1651730374-129394) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id59]} 0 0
   [junit4]   2> 3570657 INFO  (qtp1651730374-129459) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id61]} 0 0
   [junit4]   2> 3570658 INFO  (qtp1651730374-129436) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id62]} 0 0
   [junit4]   2> 3570654 INFO  (qtp1651730374-129453) [    x:00005_core] o.a.s.c.S.Request [00005_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570662 INFO  (qtp1651730374-129395) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id58]} 0 0
   [junit4]   2> 3570662 INFO  (qtp1651730374-129395) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id60]} 0 0
   [junit4]   2> 3570663 INFO  (qtp1651730374-129397) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id64]} 0 0
   [junit4]   2> 3570663 INFO  (qtp1651730374-129396) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id65]} 0 0
   [junit4]   2> 3570664 INFO  (qtp1651730374-129396) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id66]} 0 0
   [junit4]   2> 3570664 INFO  (qtp1651730374-129394) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id67]} 0 0
   [junit4]   2> 3570665 INFO  (qtp1651730374-129459) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id68]} 0 0
   [junit4]   2> 3570665 INFO  (qtp1651730374-129459) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id69]} 0 0
   [junit4]   2> 3570665 INFO  (qtp1651730374-129453) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id70]} 0 0
   [junit4]   2> 3570666 INFO  (qtp1651730374-129395) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id71]} 0 0
   [junit4]   2> 3570672 INFO  (qtp1651730374-129460) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id40]} 0 77
   [junit4]   2> 3570702 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3570702 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3570704 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.1240234375, noCFSRatio=0.6879961872112529]
   [junit4]   2> 3570705 INFO  (qtp1651730374-129393) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id73]} 0 0
   [junit4]   2> 3570705 INFO  (qtp1651730374-129396) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570705 INFO  (qtp1651730374-129393) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570705 INFO  (qtp1651730374-129396) [    x:00000_core] o.a.s.c.S.Request [00000_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570705 INFO  (qtp1651730374-129393) [    x:00005_core] o.a.s.c.S.Request [00005_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570705 INFO  (qtp1651730374-129396) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id75]} 0 0
   [junit4]   2> 3570706 INFO  (qtp1651730374-129453) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id74]} 0 0
   [junit4]   2> 3570706 INFO  (qtp1651730374-129460) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id76]} 0 0
   [junit4]   2> 3570706 INFO  (qtp1651730374-129393) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id78]} 0 0
   [junit4]   2> 3570707 INFO  (qtp1651730374-129394) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id79]} 0 0
   [junit4]   2> 3570708 INFO  (qtp1651730374-129459) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id80]} 0 0
   [junit4]   2> 3570708 INFO  (qtp1651730374-129436) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id81]} 0 0
   [junit4]   2> 3570709 INFO  (qtp1651730374-129453) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id82]} 0 0
   [junit4]   2> 3570710 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@42b79833[00004_core] main]
   [junit4]   2> 3570711 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_CD9376F4F7291608-001/tempDir-001/00004_core/conf
   [junit4]   2> 3570712 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3570740 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id72]} 0 73
   [junit4]   2> 3570740 INFO  (searcherExecutor-11771-thread-1-processing-x:00004_core) [    x:00004_core] o.a.s.c.SolrCore [00004_core] Registered new searcher Searcher@42b79833[00004_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3570740 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id18]} 0 363
   [junit4]   2> 3570740 INFO  (qtp1651730374-129397) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=36
   [junit4]   2> 3570742 INFO  (qtp1651730374-129393) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id85]} 0 0
   [junit4]   2> 3570742 INFO  (qtp1651730374-129396) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id84]} 0 0
   [junit4]   2> 3570742 INFO  (qtp1651730374-129459) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id87]} 0 0
   [junit4]   2> 3570743 INFO  (qtp1651730374-129436) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id88]} 0 0
   [junit4]   2> 3570747 INFO  (qtp1651730374-129454) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id63]} 0 88
   [junit4]   2> 3570750 INFO  (qtp1651730374-129435) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id86]} 0 8
   [junit4]   2> 3570751 INFO  (qtp1651730374-129453) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id89]} 0 7
   [junit4]   2> 3570751 INFO  (qtp1651730374-129395) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id90]} 0 0
   [junit4]   2> 3570752 INFO  (qtp1651730374-129395) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id91]} 0 0
   [junit4]   2> 3570752 INFO  (qtp1651730374-129456) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id93]} 0 0
   [junit4]   2> 3570753 INFO  (qtp1651730374-129396) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id94]} 0 0
   [junit4]   2> 3570754 INFO  (qtp1651730374-129438) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id55]} 0 98
   [junit4]   2> 3570754 INFO  (qtp1651730374-129459) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id95]} 0 0
   [junit4]   2> 3570754 INFO  (qtp1651730374-129455) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id53]} 0 100
   [junit4]   2> 3570754 INFO  (qtp1651730374-129436) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id96]} 0 0
   [junit4]   2> 3570755 INFO  (qtp1651730374-129457) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id24]} 0 322
   [junit4]   2> 3570755 INFO  (qtp1651730374-129454) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id97]} 0 0
   [junit4]   2> 3570755 INFO  (qtp1651730374-129461) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id50]} 0 116
   [junit4]   2> 3570755 INFO  (qtp1651730374-129395) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id98]} 0 0
   [junit4]   2> 3570755 INFO  (qtp1651730374-129435) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id99]} 0 0
   [junit4]   2> 3570755 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id100]} 0 0
   [junit4]   2> 3570756 INFO  (qtp1651730374-129447) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=112
   [junit4]   2> 3570756 INFO  (qtp1651730374-129456) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id101]} 0 0
   [junit4]   2> 3570756 INFO  (qtp1651730374-129448) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=103
   [junit4]   2> 3570756 INFO  (qtp1651730374-129393) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id103]} 0 0
   [junit4]   2> 3570756 INFO  (qtp1651730374-129393) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id104]} 0 0
   [junit4]   2> 3570756 INFO  (qtp1651730374-129396) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id102]} 0 0
   [junit4]   2> 3570756 INFO  (qtp1651730374-129458) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=295
   [junit4]   2> 3570757 INFO  (qtp1651730374-129459) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id105]} 0 0
   [junit4]   2> 3570757 INFO  (qtp1651730374-129458) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id106]} 0 0
   [junit4]   2> 3570757 INFO  (qtp1651730374-129459) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id107]} 0 0
   [junit4]   2> 3570757 INFO  (qtp1651730374-129458) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id108]} 0 0
   [junit4]   2> 3570758 INFO  (qtp1651730374-129395) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id110]} 0 0
   [junit4]   2> 3570758 INFO  (qtp1651730374-129395) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id111]} 0 0
   [junit4]   2> 3570759 INFO  (qtp1651730374-129395) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id112]} 0 0
   [junit4]   2> 3570759 INFO  (qtp1651730374-129397) [    x:00000_core] o.a.s.c.S.Request [00000_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570759 INFO  (qtp1651730374-129435) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id114]} 0 0
   [junit4]   2> 3570759 INFO  (qtp1651730374-129397) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id115]} 0 0
   [junit4]   2> 3570759 INFO  (qtp1651730374-129435) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id116]} 0 0
   [junit4]   2> 3570760 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id118]} 0 0
   [junit4]   2> 3570761 INFO  (qtp1651730374-129448) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id119]} 0 0
   [junit4]   2> 3570762 INFO  (qtp1651730374-129393) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id121]} 0 0
   [junit4]   2> 3570772 INFO  (qtp1651730374-129460) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id83]} 0 63
   [junit4]   2> 3570774 INFO  (qtp1651730374-129458) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id122]} 0 0
   [junit4]   2> 3570774 INFO  (qtp1651730374-129458) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id123]} 0 0
   [junit4]   2> 3570779 INFO  (qtp1651730374-129435) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id125]} 0 0
   [junit4]   2> 3570784 INFO  (qtp1651730374-129397) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id124]} 0 8
   [junit4]   2> 3570784 INFO  (qtp1651730374-129447) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id127]} 0 0
   [junit4]   2> 3570786 INFO  (qtp1651730374-129394) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id77]} 0 79
   [junit4]   2> 3570788 INFO  (qtp1651730374-129396) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id129]} 0 0
   [junit4]   2> 3570788 INFO  (qtp1651730374-129438) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id130]} 0 0
   [junit4]   2> 3570801 INFO  (qtp1651730374-129455) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id132]} 0 0
   [junit4]   2> 3570802 INFO  (qtp1651730374-129436) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id133]} 0 0
   [junit4]   2> 3570806 INFO  (qtp1651730374-129458) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570806 INFO  (qtp1651730374-129457) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570806 INFO  (qtp1651730374-129435) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570807 INFO  (qtp1651730374-129397) [    x:00000_core] o.a.s.c.S.Request [00000_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570808 INFO  (qtp1651730374-129461) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id126]} 0 28
   [junit4]   2> 3570820 INFO  (qtp1651730374-129459) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id131]} 0 30
   [junit4]   2> 3570820 INFO  (qtp1651730374-129459) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id135]} 0 0
   [junit4]   2> 3570821 INFO  (qtp1651730374-129459) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id136]} 0 0
   [junit4]   2> 3570829 INFO  (qtp1651730374-129447) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id137]} 0 0
   [junit4]   2> 3570830 INFO  (qtp1651730374-129394) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id138]} 0 0
   [junit4]   2> 3570831 INFO  (qtp1651730374-129436) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id140]} 0 0
   [junit4]   2> 3570831 INFO  (qtp1651730374-129393) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id128]} 0 46
   [junit4]   2> 3570832 INFO  (qtp1651730374-129458) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id141]} 0 0
   [junit4]   2> 3570833 INFO  (qtp1651730374-129435) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id143]} 0 0
   [junit4]   2> 3570833 INFO  (qtp1651730374-129453) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id92]} 0 81
   [junit4]   2> 3570833 INFO  (qtp1651730374-129397) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id144]} 0 0
   [junit4]   2> 3570834 INFO  (qtp1651730374-129461) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id145]} 0 0
   [junit4]   2> 3570835 INFO  (qtp1651730374-129456) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id117]} 0 75
   [junit4]   2> 3570836 INFO  (qtp1651730374-129395) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id113]} 0 76
   [junit4]   2> 3570837 INFO  (qtp1651730374-129448) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id120]} 0 76
   [junit4]   2> 3570838 INFO  (qtp1651730374-129459) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id149]} 0 0
   [junit4]   2> 3570838 INFO  (qtp1651730374-129459) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id148]} 0 0
   [junit4]   2> 3570839 INFO  (qtp1651730374-129459) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id147]} 0 0
   [junit4]   2> 3570839 INFO  (qtp1651730374-129459) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id146]} 0 0
   [junit4]   2> 3570840 INFO  (qtp1651730374-129394) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id152]} 0 0
   [junit4]   2> 3570840 INFO  (qtp1651730374-129396) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id153]} 0 0
   [junit4]   2> 3570840 INFO  (qtp1651730374-129394) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id151]} 0 0
   [junit4]   2> 3570840 INFO  (qtp1651730374-129435) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id154]} 0 0
   [junit4]   2> 3570840 INFO  (qtp1651730374-129457) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id142]} 0 8
   [junit4]   2> 3570841 INFO  (qtp1651730374-129457) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id155]} 0 0
   [junit4]   2> 3570841 INFO  (qtp1651730374-129453) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id157]} 0 0
   [junit4]   2> 3570841 INFO  (qtp1651730374-129457) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id156]} 0 0
   [junit4]   2> 3570841 INFO  (qtp1651730374-129457) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3570842 INFO  (qtp1651730374-129397) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id158]} 0 0
   [junit4]   2> 3570842 INFO  (qtp1651730374-129461) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id159]} 0 0
   [junit4]   2> 3570843 INFO  (qtp1651730374-129456) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id161]} 0 0
   [junit4]   2> 3570843 INFO  (qtp1651730374-129438) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id139]} 0 12
   [junit4]   2> 3570843 INFO  (qtp1651730374-129456) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id163]} 0 0
   [junit4]   2> 3570843 INFO  (qtp1651730374-129438) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id160]} 0 0
   [junit4]   2> 3570843 INFO  (qtp1651730374-129456) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id162]} 0 0
   [junit4]   2> 3570843 INFO  (qtp1651730374-129438) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id164]} 0 0
   [junit4]   2> 3570844 INFO  (qtp1651730374-129448) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id165]} 0 0
   [junit4]   2> 3570844 INFO  (qtp1651730374-129459) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id150]} 0 4
   [junit4]   2> 3570844 INFO  (qtp1651730374-129448) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id166]} 0 0
   [junit4]   2> 3570844 INFO  (qtp1651730374-129459) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [

[...truncated too long message...]

sTest, OverseerModifyCollectionTest, OverseerRolesTest, PeerSyncReplicationTest, RemoteQueryErrorTest, RestartWhileUpdatingTest, RollingRestartTest, ExecutePlanActionTest, NodeAddedTriggerIntegrationTest, ScheduledMaintenanceTriggerTest, TestSimNodeAddedTrigger, TestSimNodeLostTrigger, CdcrBidirectionalTest, CdcrBootstrapTest, CdcrWithNodesRestartsTest, HdfsNNFailoverTest, HdfsRecoverLeaseTest, HdfsThreadLeakTest, HdfsWriteToMultipleCollectionsTest, ZkStateWriterTest, DirectoryFactoryTest, OpenCloseCoreStressTest, DistributedFacetPivotSmallTest, ShardsWhitelistTest, TestExportWriter, RandomizedTaggerTest, HighlighterTest, TestLegacyField, TestMultiValuedNumericRangeQuery, TestNumericTokenStream, JvmMetricsTest, MetricsConfigTest, SolrCoreMetricManagerTest, SolrMetricManagerTest, SolrMetricReporterTest, SolrMetricsIntegrationTest, SolrJmxReporterTest, SolrSlf4jReporterTest, SolrRrdBackendFactoryTest, RegexBytesRefFilterTest, TestIntervalFaceting, TestRangeQuery, TestRecoveryHdfs, TestSimpleQParserPlugin, TestSolr4Spatial2, TestSolrCoreParser, TestSolrFieldCacheBean, TestSurroundQueryParser, RangeFacetCloudTest, TestScoreJoinQPScore, SimpleMLTQParserTest, TestBM25SimilarityFactory, TestPerFieldSimilarity, TestSweetSpotSimilarityFactory, TestExactSharedStatsCache, TestImpersonationWithHadoopAuth, TestZkAclsWithHadoopAuth, HttpSolrCallGetCoreTest, BlockCacheTest, BlockDirectoryTest, TestDocTermOrdsUninvertLimit, TestLegacyFieldCache, PrimUtilsTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=PrimUtilsTest -Dtests.seed=CD9376F4F7291608 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=ko -Dtests.timezone=America/Danmarkshavn -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | PrimUtilsTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [833/838 (20!)] on J1 in 1.31s, 0 tests, 1 error <<< FAILURES!

[...truncated 13 lines...]
   [junit4] Suite: org.apache.solr.util.hll.HLLSerializationTest
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false): {field=LM Jelinek-Mercer(0.100000)}, locale=ko, timezone=America/Danmarkshavn
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=59816872,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [DeleteNodeTest, TestConfigSetsAPIExclusivity, TestSubQueryTransformerCrossCore, TestReplicationHandlerBackup, TestChildDocTransformer, HdfsBasicDistributedZkTest, SolrInfoBeanTest, ExitableDirectoryReaderTest, NumberUtilsTest, RulesTest, NodeLostTriggerTest, VMParamsZkACLAndCredentialsProvidersTest, PluginInfoTest, MultiSolrCloudTestCaseTest, PrimitiveFieldTypeTest, TestLegacyFieldReuse, OverseerTaskQueueTest, TestFuzzyAnalyzedSuggestions, DisMaxRequestHandlerTest, TestSimPolicyCloud, TestSafeXMLParsing, HttpPartitionOnCommitTest, ClusterStateTest, SpellCheckCollatorTest, TestFieldCollectionResource, TestAuthorizationFramework, MetricsHistoryHandlerTest, TestMaxTokenLenTokenizer, TlogReplayBufferedWhileIndexingTest, TestIndexingPerformance, TestDFISimilarityFactory, PeerSyncTest, DirectUpdateHandlerTest, TermVectorComponentTest, SortSpecParsingTest, FastVectorHighlighterTest, TestManagedSchemaAPI, TestTlogReplica, LukeRequestHandlerTest, JsonLoaderTest, TestExpandComponent, DirectSolrSpellCheckerTest, MergeStrategyTest, DistribJoinFromCollectionTest, RequestHandlersTest, ActionThrottleTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, AutoAddReplicasPlanActionTest, TestTrieFacet, TestSlowCompositeReaderWrapper, SolrCoreCheckLockOnStartupTest, IndexSizeTriggerTest, SubstringBytesRefFilterTest, TestRequestForwarding, TestSearchPerf, SpellingQueryConverterTest, TestPolicyCloud, TestEmbeddedSolrServerSchemaAPI, ZkFailoverTest, LIRRollingUpdatesTest, TestBlendedInfixSuggestions, TriggerIntegrationTest, TestElisionMultitermQuery, ExplicitHLLTest, DistributedMLTComponentTest, TestMissingGroups, TestRawTransformer, CollectionTooManyReplicasTest, TestLazyCores, TestBadConfig, TestDistributedSearch, ResponseLogComponentTest, MetricsHistoryIntegrationTest, PolyFieldTest, NotRequiredUniqueKeyTest, TestSha256AuthenticationProvider, TestPrepRecovery, CdcrReplicationHandlerTest, CurrencyRangeFacetCloudTest, TestInfoStreamLogging, BigEndianAscendingWordDeserializerTest, TestPointFields, TaggerTest, SearchRateTriggerIntegrationTest, DistributedUpdateProcessorTest, CSVRequestHandlerTest, StressHdfsTest, ConfigureRecoveryStrategyTest, HdfsTlogReplayBufferedWhileIndexingTest, BadComponentTest, TestSolrQueryParser, SpatialHeatmapFacetsTest, ProtectedTermFilterFactoryTest, TestUniqueKeyFieldResource, TestCopyFieldCollectionResource, BasicDistributedZkTest, RequestLoggingTest, TestJsonFacetRefinement, TestCloudManagedSchema, TestFieldCacheSort, TestLMDirichletSimilarityFactory, CollectionStateFormat2Test, TestStressLucene, TestBlobHandler, TestFieldTypeResource, TestCloudDeleteByQuery, UtilsToolTest, TestHighlightDedupGrouping, TestStressUserVersions, TestBinaryResponseWriter, ClassificationUpdateProcessorTest, PeerSyncWithLeaderAndIndexFingerprintCachingTest, V2StandaloneTest, TestPHPSerializedResponseWriter, TestStressCloudBlindAtomicUpdates, TestSchemaSimilarityResource, TestDeleteCollectionOnDownNodes, BasicAuthIntegrationTest, TestConfigSets, ResponseHeaderTest, TestJsonFacetsWithNestedObjects, TestCollapseQParserPlugin, TestCoreBackup, AnalysisErrorHandlingTest, CreateRoutedAliasTest, DateMathParserTest, ConfigSetsAPITest, UpdateRequestProcessorFactoryTest, SolrIndexSplitterTest, TestDynamicLoading, CurrencyFieldTypeTest, ReplaceNodeNoTargetTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestManagedResourceStorage, SolrGraphiteReporterTest, SuggesterWFSTTest, TestSmileRequest, DistributedQueryElevationComponentTest, AssignBackwardCompatibilityTest, AsyncCallRequestStatusResponseTest, BasicDistributedZk2Test, DocValuesNotIndexedTest, ForceLeaderTest, LeaderElectionIntegrationTest, LeaderElectionTest, LeaderFailoverAfterPartitionTest, LeaderTragicEventTest, LeaderVoteWaitTimeoutTest, MoveReplicaHDFSTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerModifyCollectionTest, OverseerRolesTest, PeerSyncReplicationTest, RemoteQueryErrorTest, RestartWhileUpdatingTest, RollingRestartTest, ExecutePlanActionTest, NodeAddedTriggerIntegrationTest, ScheduledMaintenanceTriggerTest, TestSimNodeAddedTrigger, TestSimNodeLostTrigger, CdcrBidirectionalTest, CdcrBootstrapTest, CdcrWithNodesRestartsTest, HdfsNNFailoverTest, HdfsRecoverLeaseTest, HdfsThreadLeakTest, HdfsWriteToMultipleCollectionsTest, ZkStateWriterTest, DirectoryFactoryTest, OpenCloseCoreStressTest, DistributedFacetPivotSmallTest, ShardsWhitelistTest, TestExportWriter, RandomizedTaggerTest, HighlighterTest, TestLegacyField, TestMultiValuedNumericRangeQuery, TestNumericTokenStream, JvmMetricsTest, MetricsConfigTest, SolrCoreMetricManagerTest, SolrMetricManagerTest, SolrMetricReporterTest, SolrMetricsIntegrationTest, SolrJmxReporterTest, SolrSlf4jReporterTest, SolrRrdBackendFactoryTest, RegexBytesRefFilterTest, TestIntervalFaceting, TestRangeQuery, TestRecoveryHdfs, TestSimpleQParserPlugin, TestSolr4Spatial2, TestSolrCoreParser, TestSolrFieldCacheBean, TestSurroundQueryParser, RangeFacetCloudTest, TestScoreJoinQPScore, SimpleMLTQParserTest, TestBM25SimilarityFactory, TestPerFieldSimilarity, TestSweetSpotSimilarityFactory, TestExactSharedStatsCache, TestImpersonationWithHadoopAuth, TestZkAclsWithHadoopAuth, HttpSolrCallGetCoreTest, BlockCacheTest, BlockDirectoryTest, TestDocTermOrdsUninvertLimit, TestLegacyFieldCache, PrimUtilsTest, HLLSerializationTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HLLSerializationTest -Dtests.seed=CD9376F4F7291608 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=ko -Dtests.timezone=America/Danmarkshavn -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | HLLSerializationTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Completed [838/838 (21!)] on J1 in 2.00s, 0 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/temp/junit4-J1-20190208_233745_2535371001509277691787.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/heapdumps/java_pid26075.hprof ...
   [junit4] Heap dump file created [457158585 bytes in 2.348 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 8916 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:651: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid26075.hprof

Total time: 289 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 8 - Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/8/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, InternalHttpClient, MMapDirectory, SolrCore, MMapDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:503)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:346)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:425)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1171)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:225)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:267)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:421)  at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:237)  at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:382)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:328)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:307)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:770)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1054)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:359)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:738)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)   expected null, but was:<ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, InternalHttpClient, MMapDirectory, SolrCore, MMapDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:503)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:346)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:425)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1171)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:225)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:267)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:421)  at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:237)  at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:382)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:328)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:307)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:770)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1054)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:359)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:738)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  >

Stack Trace:
java.lang.AssertionError: ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, InternalHttpClient, MMapDirectory, SolrCore, MMapDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:503)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:346)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:425)
        at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1171)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
        at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:225)
        at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:267)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:421)
        at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:237)
        at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:382)
        at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:328)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:307)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:770)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1054)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:359)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:738)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

 expected null, but was:<ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, InternalHttpClient, MMapDirectory, SolrCore, MMapDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:503)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:346)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:425)
        at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1171)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
        at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:225)
        at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:267)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:421)
        at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:237)
        at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:382)
        at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:328)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:307)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:770)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1054)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:359)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:738)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

>
        at __randomizedtesting.SeedInfo.seed([553F8366882B6061]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotNull(Assert.java:755)
        at org.junit.Assert.assertNull(Assert.java:737)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:320)
        at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testAboveSearchRate

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([553F8366882B6061:A3343EA0624F5ABD]:0)
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertTrue(Assert.java:52)
        at org.apache.solr.cloud.autoscaling.SearchRateTriggerIntegrationTest.testAboveSearchRate(SearchRateTriggerIntegrationTest.java:265)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.HdfsTransactionLog  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)  at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:76)  at org.apache.solr.update.HdfsUpdateLog.ensureLog(HdfsUpdateLog.java:335)  at org.apache.solr.update.UpdateLog.deleteByQuery(UpdateLog.java:660)  at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:531)  at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:78)  at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:59)  at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:1007)  at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDeleteByQuery(DistributedUpdateProcessor.java:1744)  at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1649)  at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1434)  at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:124)  at org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:210)  at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:128)  at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)  at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)  at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)  at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)   expected null, but was:<ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.HdfsTransactionLog  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)  at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:76)  at org.apache.solr.update.HdfsUpdateLog.ensureLog(HdfsUpdateLog.java:335)  at org.apache.solr.update.UpdateLog.deleteByQuery(UpdateLog.java:660)  at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:531)  at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:78)  at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:59)  at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:1007)  at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDeleteByQuery(DistributedUpdateProcessor.java:1744)  at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1649)  at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1434)  at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:124)  at org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:210)  at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:128)  at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)  at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)  at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)  at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  >

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.HdfsTransactionLog
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)
        at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:76)
        at org.apache.solr.update.HdfsUpdateLog.ensureLog(HdfsUpdateLog.java:335)
        at org.apache.solr.update.UpdateLog.deleteByQuery(UpdateLog.java:660)
        at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:531)
        at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:78)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:59)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:1007)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDeleteByQuery(DistributedUpdateProcessor.java:1744)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1649)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1434)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:124)
        at org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:210)
        at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:128)
        at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

 expected null, but was:<ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.HdfsTransactionLog
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)
        at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:76)
        at org.apache.solr.update.HdfsUpdateLog.ensureLog(HdfsUpdateLog.java:335)
        at org.apache.solr.update.UpdateLog.deleteByQuery(UpdateLog.java:660)
        at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:531)
        at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:78)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:59)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:1007)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDeleteByQuery(DistributedUpdateProcessor.java:1744)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1649)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1434)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:124)
        at org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:210)
        at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:128)
        at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

>
        at __randomizedtesting.SeedInfo.seed([553F8366882B6061]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotNull(Assert.java:755)
        at org.junit.Assert.assertNull(Assert.java:737)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:320)
        at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Error from server at http://127.0.0.1:44028/delete_data_dir: Expected mime type application/octet-stream but got application/json. {   "error":{     "metadata":[       "error-class","org.apache.solr.common.SolrException",       "root-error-class","java.net.SocketTimeoutException"],     "msg":"Error trying to proxy request for url: http://127.0.0.1:38608/delete_data_dir/update",     "trace":"org.apache.solr.common.SolrException: Error trying to proxy request for url: http://127.0.0.1:38608/delete_data_dir/update\n\tat org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:646)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:500)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.net.SocketTimeoutException: Read timed out\n\tat java.net.SocketInputStream.socketRead0(Native Method)\n\tat java.net.SocketInputStream.socketRead(SocketInputStream.java:116)\n\tat java.net.SocketInputStream.read(SocketInputStream.java:171)\n\tat java.net.SocketInputStream.read(SocketInputStream.java:141)\n\tat org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)\n\tat org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)\n\tat org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)\n\tat org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)\n\tat org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)\n\tat org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)\n\tat org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)\n\tat org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)\n\tat org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)\n\tat org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)\n\tat org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:120)\n\tat org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)\n\tat org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)\n\tat org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)\n\tat org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)\n\tat org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)\n\tat org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:618)\n\t... 28 more\n",     "code":500}}

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:44028/delete_data_dir: Expected mime type application/octet-stream but got application/json. {
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","java.net.SocketTimeoutException"],
    "msg":"Error trying to proxy request for url: http://127.0.0.1:38608/delete_data_dir/update",
    "trace":"org.apache.solr.common.SolrException: Error trying to proxy request for url: http://127.0.0.1:38608/delete_data_dir/update\n\tat org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:646)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:500)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.net.SocketTimeoutException: Read timed out\n\tat java.net.SocketInputStream.socketRead0(Native Method)\n\tat java.net.SocketInputStream.socketRead(SocketInputStream.java:116)\n\tat java.net.SocketInputStream.read(SocketInputStream.java:171)\n\tat java.net.SocketInputStream.read(SocketInputStream.java:141)\n\tat org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)\n\tat org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)\n\tat org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)\n\tat org.apache.h

[...truncated too long message...]

urce] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null478873447
     [copy] Copying 242 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null478873447
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null478873447

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 147ms :: artifacts dl 20ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 487 minutes 41 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 9 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/9/

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, MMapDirectory, MMapDirectory, SolrCore, InternalHttpClient] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:359)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:738)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:503)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:346)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:425)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1171)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:508)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:959)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1054)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:225)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:267)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:421)  at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:237)  at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:382)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:328)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:307)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)   expected null, but was:<ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, MMapDirectory, MMapDirectory, SolrCore, InternalHttpClient] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:359)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:738)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:503)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:346)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:425)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1171)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:508)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:959)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1054)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)  at org.eclipse.jetty.server.Server.handle(Server.java:502)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:225)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:267)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:421)  at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:237)  at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:382)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:328)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:307)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  >

Stack Trace:
java.lang.AssertionError: ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, MMapDirectory, MMapDirectory, SolrCore, InternalHttpClient]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:359)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:738)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:503)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:346)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:425)
        at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1171)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:508)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:959)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1054)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
        at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:225)
        at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:267)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:421)
        at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:237)
        at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:382)
        at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:328)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:307)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

 expected null, but was:<ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, MMapDirectory, MMapDirectory, SolrCore, InternalHttpClient]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:359)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:738)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:967)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:503)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:346)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:425)
        at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1171)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:508)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:959)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1054)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:874)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:502)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
        at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:225)
        at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:267)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:421)
        at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:237)
        at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:382)
        at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:328)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:307)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

>
        at __randomizedtesting.SeedInfo.seed([1D3EFC051EE42746]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotNull(Assert.java:755)
        at org.junit.Assert.assertNull(Assert.java:737)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:320)
        at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 13285 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_1D3EFC051EE42746-001/init-core-data-001
   [junit4]   2> 287786 WARN  (SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[1D3EFC051EE42746]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=8 numCloses=8
   [junit4]   2> 287786 INFO  (SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[1D3EFC051EE42746]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 287788 INFO  (SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[1D3EFC051EE42746]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 287788 INFO  (SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[1D3EFC051EE42746]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 287789 INFO  (SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[1D3EFC051EE42746]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 287791 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ChaosMonkeyNothingIsSafeWithPullReplicasTest Starting ChaosMonkey test with 1 shards and 3 nodes
   [junit4]   2> 287794 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 287794 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 287794 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 287894 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer start zk server on port:40581
   [junit4]   2> 287894 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:40581
   [junit4]   2> 287894 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 40581
   [junit4]   2> 287902 INFO  (zkConnectionManagerCallback-1252-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 287905 INFO  (zkConnectionManagerCallback-1254-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 287907 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 287908 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 287910 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 287911 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 287913 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 287914 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 287916 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 287917 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 287918 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 287920 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 287921 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkTestServer put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 287922 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 288091 WARN  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 288092 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 288092 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 288092 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 288093 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 288093 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 288093 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 288094 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7e6de9f{/,null,AVAILABLE}
   [junit4]   2> 288095 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7f0be9b2{HTTP/1.1,[http/1.1]}{127.0.0.1:42451}
   [junit4]   2> 288095 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.e.j.s.Server Started @288150ms
   [junit4]   2> 288095 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_1D3EFC051EE42746-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=42451, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_1D3EFC051EE42746-001/control-001/cores}
   [junit4]   2> 288095 ERROR (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 288095 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 288096 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 288096 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 288096 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 288096 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-12T06:16:16.504Z
   [junit4]   2> 288098 INFO  (zkConnectionManagerCallback-1256-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 288099 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 288099 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_1D3EFC051EE42746-001/control-001/solr.xml
   [junit4]   2> 288104 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 288104 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 288106 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@59c82380, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 288232 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 288237 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40581/solr
   [junit4]   2> 288239 INFO  (zkConnectionManagerCallback-1260-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 288241 INFO  (zkConnectionManagerCallback-1262-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 288329 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [n:127.0.0.1:42451_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42451_
   [junit4]   2> 288329 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [n:127.0.0.1:42451_    ] o.a.s.c.Overseer Overseer (id=72784187550269444-127.0.0.1:42451_-n_0000000000) starting
   [junit4]   2> 288337 INFO  (zkConnectionManagerCallback-1269-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 288339 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [n:127.0.0.1:42451_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40581/solr ready
   [junit4]   2> 288341 INFO  (OverseerStateUpdate-72784187550269444-127.0.0.1:42451_-n_0000000000) [n:127.0.0.1:42451_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:42451_
   [junit4]   2> 288341 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [n:127.0.0.1:42451_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42451_
   [junit4]   2> 288347 INFO  (zkCallback-1268-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 288347 INFO  (OverseerStateUpdate-72784187550269444-127.0.0.1:42451_-n_0000000000) [n:127.0.0.1:42451_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 288382 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [n:127.0.0.1:42451_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 288408 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [n:127.0.0.1:42451_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59c82380
   [junit4]   2> 288421 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [n:127.0.0.1:42451_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59c82380
   [junit4]   2> 288421 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [n:127.0.0.1:42451_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59c82380
   [junit4]   2> 288423 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [n:127.0.0.1:42451_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_1D3EFC051EE42746-001/control-001/cores
   [junit4]   2> 288456 INFO  (zkConnectionManagerCallback-1275-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 288457 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 288458 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40581/solr ready
   [junit4]   2> 288462 INFO  (qtp1033049173-4377) [n:127.0.0.1:42451_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42451_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 288466 INFO  (OverseerThreadFactory-1983-thread-1-processing-n:127.0.0.1:42451_) [n:127.0.0.1:42451_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 288573 INFO  (qtp1033049173-4379) [n:127.0.0.1:42451_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 288574 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 288576 INFO  (qtp1033049173-4376) [n:127.0.0.1:42451_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 288577 INFO  (qtp1033049173-4378) [n:127.0.0.1:42451_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 288578 INFO  (qtp1033049173-4379) [n:127.0.0.1:42451_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 288581 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 288582 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 289610 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.1
   [junit4]   2> 289644 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 289750 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 289793 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 289793 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@59c82380
   [junit4]   2> 289794 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_1D3EFC051EE42746-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_1D3EFC051EE42746-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 289799 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=8, maxMergeAtOnceExplicit=2, maxMergedSegmentMB=21.4130859375, floorSegmentMB=0.87109375, forceMergeDeletesPctAllowed=21.029578914370337, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=26.382491234975436
   [junit4]   2> 289817 WARN  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 289892 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 289892 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 289914 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
   [junit4]   2> 289914 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 289917 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13262337761742238]
   [junit4]   2> 289918 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@562f548b[control_collection_shard1_replica_n1] main]
   [junit4]   2> 289919 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 289920 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 289921 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 289921 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1625242655343509504
   [junit4]   2> 289925 INFO  (searcherExecutor-1988-thread-1-processing-n:127.0.0.1:42451_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@562f548b[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 289928 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 289928 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 289933 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 289933 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 289933 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:42451/control_collection_shard1_replica_n1/
   [junit4]   2> 289933 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 289934 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:42451/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 289934 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 289935 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/72784187550269444-core_node2-n_0000000000
   [junit4]   2> 289938 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:42451/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 290040 INFO  (zkCallback-1261-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 290041 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 290044 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1462
   [junit4]   2> 290052 INFO  (qtp1033049173-4377) [n:127.0.0.1:42451_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 290147 INFO  (zkCallback-1261-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 290147 INFO  (zkCallback-1261-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 290153 INFO  (qtp1033049173-4377) [n:127.0.0.1:42451_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42451_&wt=javabin&version=2} status=0 QTime=1690
   [junit4]   2> 290161 INFO  (zkConnectionManagerCallback-1280-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 290162 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 290163 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40581/solr ready
   [junit4]   2> 290163 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 290165 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 290171 INFO  (OverseerThreadFactory-1983-thread-2-processing-n:127.0.0.1:42451_) [n:127.0.0.1:42451_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 290173 INFO  (OverseerCollectionConfigSetProcessor-72784187550269444-127.0.0.1:42451_-n_0000000000) [n:127.0.0.1:42451_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 290378 WARN  (OverseerThreadFactory-1983-thread-2-processing-n:127.0.0.1:42451_) [n:127.0.0.1:42451_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 290381 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 290383 INFO  (qtp1033049173-4380) [n:127.0.0.1:42451_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=218
   [junit4]   2> 290385 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=1 numOtherReplicas=2
   [junit4]   2> 290568 INFO  (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[1D3EFC051EE42746]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_1D3EFC051EE42746-001/shard-1-001 of type NRT
   [junit4]   2> 290591 WARN  (closeThreadPool-1281-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 290591 INFO  (closeThreadPool-1281-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 290591 INFO  (closeThreadPool-1281-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 290592 INFO  (closeThreadPool-1281-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 290597 INFO  (closeThreadPool-1281-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 290597 INFO  (closeThreadPool-1281-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 290597 INFO  (closeThreadPool-1281-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 290598 INFO  (closeThreadPool-1281-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@beee22e{/,null,AVAILABLE}
   [junit4]   2> 290599 INFO  (closeThreadPool-1281-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@196f1f7c{HTTP/1.1,[http/1.1]}{127.0.0

[...truncated too long message...]



ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null126049477
     [copy] Copying 242 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null126049477
   [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null126049477

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 51ms :: artifacts dl 4ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 260 minutes 49 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 10 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/10/

1 tests failed.
FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
Some docs had errors -- check logs expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: Some docs had errors -- check logs expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([99829954D323FA78:AF96FB12597EC069]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:834)
        at org.junit.Assert.assertEquals(Assert.java:645)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:352)
        at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:207)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14911 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/init-core-data-001
   [junit4]   2> 2600566 WARN  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 2600567 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2600578 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=SSL overhead seems to cause OutOfMemory when stress testing)
   [junit4]   2> 2600579 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2600582 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 10 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001
   [junit4]   2> 2600583 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2600585 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2600585 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2600685 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:43098
   [junit4]   2> 2600685 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43098
   [junit4]   2> 2600685 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 43098
   [junit4]   2> 2600689 INFO  (zkConnectionManagerCallback-8166-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600693 INFO  (zkConnectionManagerCallback-8168-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600695 INFO  (zkConnectionManagerCallback-8170-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600698 WARN  (jetty-launcher-8171-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2600698 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2600698 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2600698 INFO  (jetty-launcher-8171-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2600699 WARN  (jetty-launcher-8171-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2600699 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2600699 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2600699 INFO  (jetty-launcher-8171-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2600700 WARN  (jetty-launcher-8171-thread-3) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2600701 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2600701 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2600701 INFO  (jetty-launcher-8171-thread-3) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2600701 WARN  (jetty-launcher-8171-thread-5) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2600701 WARN  (jetty-launcher-8171-thread-6) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2600702 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2600702 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2600702 INFO  (jetty-launcher-8171-thread-5) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2600702 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2600702 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2600702 INFO  (jetty-launcher-8171-thread-6) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2600702 WARN  (jetty-launcher-8171-thread-7) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2600703 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2600703 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2600703 INFO  (jetty-launcher-8171-thread-7) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2600703 WARN  (jetty-launcher-8171-thread-8) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2600703 WARN  (jetty-launcher-8171-thread-9) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2600704 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2600704 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2600704 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2600704 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2600704 INFO  (jetty-launcher-8171-thread-8) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2600704 INFO  (jetty-launcher-8171-thread-9) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2600704 WARN  (jetty-launcher-8171-thread-4) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2600705 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2600705 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2600705 INFO  (jetty-launcher-8171-thread-4) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2600705 WARN  (jetty-launcher-8171-thread-10) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 2600705 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 2600705 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2600705 INFO  (jetty-launcher-8171-thread-10) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 2600706 INFO  (jetty-launcher-8171-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2600706 INFO  (jetty-launcher-8171-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2600706 INFO  (jetty-launcher-8171-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2600707 INFO  (jetty-launcher-8171-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@367b7a6c{/solr,null,AVAILABLE}
   [junit4]   2> 2600709 INFO  (jetty-launcher-8171-thread-4) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2600709 INFO  (jetty-launcher-8171-thread-4) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2600709 INFO  (jetty-launcher-8171-thread-4) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2600709 INFO  (jetty-launcher-8171-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3fdf45ab{/solr,null,AVAILABLE}
   [junit4]   2> 2600711 INFO  (jetty-launcher-8171-thread-9) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2600711 INFO  (jetty-launcher-8171-thread-9) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2600711 INFO  (jetty-launcher-8171-thread-9) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2600711 INFO  (jetty-launcher-8171-thread-9) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2b2f0d29{/solr,null,AVAILABLE}
   [junit4]   2> 2600715 INFO  (jetty-launcher-8171-thread-6) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2600715 INFO  (jetty-launcher-8171-thread-6) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2600715 INFO  (jetty-launcher-8171-thread-6) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2600715 INFO  (jetty-launcher-8171-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@168e5e8{/solr,null,AVAILABLE}
   [junit4]   2> 2600717 INFO  (jetty-launcher-8171-thread-5) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2600717 INFO  (jetty-launcher-8171-thread-5) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2600717 INFO  (jetty-launcher-8171-thread-5) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2600717 INFO  (jetty-launcher-8171-thread-7) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2600717 INFO  (jetty-launcher-8171-thread-7) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2600717 INFO  (jetty-launcher-8171-thread-7) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2600717 INFO  (jetty-launcher-8171-thread-7) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@664d96d3{/solr,null,AVAILABLE}
   [junit4]   2> 2600718 INFO  (jetty-launcher-8171-thread-8) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2600718 INFO  (jetty-launcher-8171-thread-8) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2600718 INFO  (jetty-launcher-8171-thread-8) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2600718 INFO  (jetty-launcher-8171-thread-8) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@630bebe8{/solr,null,AVAILABLE}
   [junit4]   2> 2600719 INFO  (jetty-launcher-8171-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2600719 INFO  (jetty-launcher-8171-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2600719 INFO  (jetty-launcher-8171-thread-1) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2600720 INFO  (jetty-launcher-8171-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@90146b8{/solr,null,AVAILABLE}
   [junit4]   2> 2600720 INFO  (jetty-launcher-8171-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@eb58717{HTTP/1.1,[http/1.1]}{127.0.0.1:38131}
   [junit4]   2> 2600720 INFO  (jetty-launcher-8171-thread-4) [    ] o.e.j.s.Server Started @2600790ms
   [junit4]   2> 2600720 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38131}
   [junit4]   2> 2600720 INFO  (jetty-launcher-8171-thread-9) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3cfb4a45{HTTP/1.1,[http/1.1]}{127.0.0.1:43002}
   [junit4]   2> 2600720 INFO  (jetty-launcher-8171-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@401d36ca{HTTP/1.1,[http/1.1]}{127.0.0.1:36671}
   [junit4]   2> 2600720 INFO  (jetty-launcher-8171-thread-1) [    ] o.e.j.s.Server Started @2600790ms
   [junit4]   2> 2600720 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36671}
   [junit4]   2> 2600720 ERROR (jetty-launcher-8171-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2600720 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2600721 ERROR (jetty-launcher-8171-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2600721 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2600721 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 2600721 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 2600721 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2600721 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2600721 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2600721 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2600721 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-14T01:01:18.549Z
   [junit4]   2> 2600721 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-14T01:01:18.549Z
   [junit4]   2> 2600720 INFO  (jetty-launcher-8171-thread-9) [    ] o.e.j.s.Server Started @2600790ms
   [junit4]   2> 2600722 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43002}
   [junit4]   2> 2600722 ERROR (jetty-launcher-8171-thread-9) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2600722 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2600722 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 2600722 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2600722 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2600722 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-14T01:01:18.550Z
   [junit4]   2> 2600723 INFO  (jetty-launcher-8171-thread-10) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2600723 INFO  (jetty-launcher-8171-thread-10) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2600723 INFO  (jetty-launcher-8171-thread-10) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2600723 INFO  (jetty-launcher-8171-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@617579ea{/solr,null,AVAILABLE}
   [junit4]   2> 2600724 INFO  (jetty-launcher-8171-thread-10) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@141b2ecc{/solr,null,AVAILABLE}
   [junit4]   2> 2600724 INFO  (jetty-launcher-8171-thread-6) [    ] o.e.j.s.AbstractConnector Started ServerConnector@728bb7af{HTTP/1.1,[http/1.1]}{127.0.0.1:36836}
   [junit4]   2> 2600724 INFO  (jetty-launcher-8171-thread-6) [    ] o.e.j.s.Server Started @2600794ms
   [junit4]   2> 2600724 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36836}
   [junit4]   2> 2600724 ERROR (jetty-launcher-8171-thread-6) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2600724 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2600724 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 2600724 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2600724 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2600725 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-14T01:01:18.553Z
   [junit4]   2> 2600724 INFO  (jetty-launcher-8171-thread-5) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5c597b0a{HTTP/1.1,[http/1.1]}{127.0.0.1:43136}
   [junit4]   2> 2600725 INFO  (jetty-launcher-8171-thread-5) [    ] o.e.j.s.Server Started @2600795ms
   [junit4]   2> 2600725 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43136}
   [junit4]   2> 2600725 ERROR (jetty-launcher-8171-thread-5) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2600725 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2600725 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 2600725 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2600726 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2600726 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-14T01:01:18.554Z
   [junit4]   2> 2600727 INFO  (jetty-launcher-8171-thread-3) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2600727 INFO  (jetty-launcher-8171-thread-3) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2600727 INFO  (jetty-launcher-8171-thread-3) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2600732 INFO  (jetty-launcher-8171-thread-8) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6f44757b{HTTP/1.1,[http/1.1]}{127.0.0.1:36980}
   [junit4]   2> 2600732 INFO  (jetty-launcher-8171-thread-8) [    ] o.e.j.s.Server Started @2600802ms
   [junit4]   2> 2600732 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36980}
   [junit4]   2> 2600732 INFO  (jetty-launcher-8171-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2d9f81cd{HTTP/1.1,[http/1.1]}{127.0.0.1:44645}
   [junit4]   2> 2600732 INFO  (jetty-launcher-8171-thread-2) [    ] o.e.j.s.Server Started @2600802ms
   [junit4]   2> 2600732 ERROR (jetty-launcher-8171-thread-8) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2600732 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44645}
   [junit4]   2> 2600732 ERROR (jetty-launcher-8171-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2600732 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2600733 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 2600733 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2600733 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2600733 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-14T01:01:18.561Z
   [junit4]   2> 2600732 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2600733 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 2600733 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2600733 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2600733 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-14T01:01:18.561Z
   [junit4]   2> 2600734 INFO  (zkConnectionManagerCallback-8174-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600735 INFO  (jetty-launcher-8171-thread-7) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5ee92fea{HTTP/1.1,[http/1.1]}{127.0.0.1:45731}
   [junit4]   2> 2600735 INFO  (jetty-launcher-8171-thread-7) [    ] o.e.j.s.Server Started @2600805ms
   [junit4]   2> 2600735 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45731}
   [junit4]   2> 2600735 ERROR (jetty-launcher-8171-thread-7) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2600735 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2600735 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 2600735 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2600735 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2600736 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-14T01:01:18.564Z
   [junit4]   2> 2600736 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2600736 INFO  (zkConnectionManagerCallback-8175-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600738 INFO  (jetty-launcher-8171-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7500a14b{/solr,null,AVAILABLE}
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-10) [    ] o.e.j.s.AbstractConnector Started ServerConnector@52eee9be{HTTP/1.1,[http/1.1]}{127.0.0.1:32952}
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-10) [    ] o.e.j.s.Server Started @2600809ms
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=32952}
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@39e234f7{HTTP/1.1,[http/1.1]}{127.0.0.1:42771}
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-3) [    ] o.e.j.s.Server Started @2600809ms
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42771}
   [junit4]   2> 2600739 ERROR (jetty-launcher-8171-thread-10) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 2600739 ERROR (jetty-launcher-8171-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-14T01:01:18.567Z
   [junit4]   2> 2600741 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2600739 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.7.1
   [junit4]   2> 2600747 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2600747 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2600747 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-14T01:01:18.575Z
   [junit4]   2> 2600754 INFO  (zkConnectionManagerCallback-8177-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600754 INFO  (zkConnectionManagerCallback-8185-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600757 INFO  (zkConnectionManagerCallback-8181-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600758 INFO  (zkConnectionManagerCallback-8187-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600758 INFO  (zkConnectionManagerCallback-8179-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600758 INFO  (zkConnectionManagerCallback-8183-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600761 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2600761 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2600761 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2600762 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2600762 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2600762 INFO  (zkConnectionManagerCallback-8191-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600763 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2600764 INFO  (zkConnectionManagerCallback-8189-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600764 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2600772 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2600780 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1029e0198ba0007, likely client has closed socket
   [junit4]   2> 2600780 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1029e0198ba000a, likely client has closed socket
   [junit4]   2> 2600807 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1029e0198ba0003, likely client has closed socket
   [junit4]   2> 2600807 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1029e0198ba0006, likely client has closed socket
   [junit4]   2> 2600958 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2600966 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2600971 INFO  (jetty-launcher-8171-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43098/solr
   [junit4]   2> 2600973 INFO  (zkConnectionManagerCallback-8195-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600975 INFO  (zkConnectionManagerCallback-8198-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600976 INFO  (jetty-launcher-8171-thread-9) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43098/solr
   [junit4]   2> 2600980 INFO  (zkConnectionManagerCallback-8201-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2600982 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1029e0198ba000f, likely client has closed socket
   [junit4]   2> 2601000 INFO  (zkConnectionManagerCallback-8203-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601038 INFO  (jetty-launcher-8171-thread-9) [n:127.0.0.1:43002_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43002_solr
   [junit4]   2> 2601040 INFO  (jetty-launcher-8171-thread-9) [n:127.0.0.1:43002_solr    ] o.a.s.c.Overseer Overseer (id=72794273685831696-127.0.0.1:43002_solr-n_0000000000) starting
   [junit4]   2> 2601072 INFO  (zkConnectionManagerCallback-8212-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601078 INFO  (jetty-launcher-8171-thread-9) [n:127.0.0.1:43002_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2601086 INFO  (OverseerStateUpdate-72794273685831696-127.0.0.1:43002_solr-n_0000000000) [n:127.0.0.1:43002_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:43002_solr
   [junit4]   2> 2601095 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2601097 INFO  (jetty-launcher-8171-thread-9) [n:127.0.0.1:43002_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43002_solr
   [junit4]   2> 2601102 INFO  (zkCallback-8211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2601116 INFO  (zkCallback-8202-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2601142 INFO  (jetty-launcher-8171-thread-9) [n:127.0.0.1:43002_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2601157 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2601169 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:38131_solr as DOWN
   [junit4]   2> 2601170 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2601170 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38131_solr
   [junit4]   2> 2601194 INFO  (jetty-launcher-8171-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43098/solr
   [junit4]   2> 2601201 INFO  (zkCallback-8211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2601202 INFO  (zkCallback-8197-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2601204 INFO  (zkCallback-8202-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2601205 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2601211 INFO  (jetty-launcher-8171-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43098/solr
   [junit4]   2> 2601211 INFO  (zkConnectionManagerCallback-8217-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601265 INFO  (zkConnectionManagerCallback-8221-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601288 INFO  (zkConnectionManagerCallback-8223-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601290 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2601300 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2601305 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:44645_solr as DOWN
   [junit4]   2> 2601305 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2601305 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44645_solr
   [junit4]   2> 2601306 INFO  (jetty-launcher-8171-thread-10) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43098/solr
   [junit4]   2> 2601307 INFO  (zkCallback-8202-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2601307 INFO  (zkCallback-8211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2601315 INFO  (zkCallback-8197-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2601324 INFO  (jetty-launcher-8171-thread-9) [n:127.0.0.1:43002_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43002.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601326 INFO  (zkConnectionManagerCallback-8225-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601329 INFO  (zkConnectionManagerCallback-8231-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601337 INFO  (zkCallback-8222-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2601364 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2601366 INFO  (zkConnectionManagerCallback-8236-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601368 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2601368 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:36836_solr as DOWN
   [junit4]   2> 2601369 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2601369 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36836_solr
   [junit4]   2> 2601369 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2601370 INFO  (zkCallback-8202-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2601370 INFO  (zkCallback-8211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2601370 INFO  (zkCallback-8197-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2601373 INFO  (jetty-launcher-8171-thread-9) [n:127.0.0.1:43002_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43002.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601374 INFO  (jetty-launcher-8171-thread-9) [n:127.0.0.1:43002_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43002.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601375 INFO  (jetty-launcher-8171-thread-9) [n:127.0.0.1:43002_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001/node9/.
   [junit4]   2> 2601384 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2601404 INFO  (zkCallback-8222-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2601441 INFO  (zkConnectionManagerCallback-8241-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601468 INFO  (zkCallback-8224-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2601469 INFO  (zkCallback-8235-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2601471 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2601481 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2601487 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38131.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601488 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:32952_solr as DOWN
   [junit4]   2> 2601489 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2601489 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32952_solr
   [junit4]   2> 2601491 INFO  (zkCallback-8222-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2601491 INFO  (zkCallback-8197-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2601492 INFO  (zkCallback-8202-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2601492 INFO  (zkCallback-8211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2601507 INFO  (zkCallback-8235-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2601530 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38131.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601531 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38131.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601533 INFO  (zkCallback-8224-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2601535 INFO  (jetty-launcher-8171-thread-4) [n:127.0.0.1:38131_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001/node4/.
   [junit4]   2> 2601552 INFO  (zkCallback-8240-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 2601558 INFO  (jetty-launcher-8171-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43098/solr
   [junit4]   2> 2601590 INFO  (zkConnectionManagerCallback-8254-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601592 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2601593 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2601604 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2601638 INFO  (zkConnectionManagerCallback-8257-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601670 INFO  (zkConnectionManagerCallback-8249-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601672 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2601674 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2601694 INFO  (zkConnectionManagerCallback-8262-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601695 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2601697 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2601702 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44645.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601708 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2601732 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2601742 INFO  (zkConnectionManagerCallback-8265-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601747 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 2601751 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:43136_solr as DOWN
   [junit4]   2> 2601752 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2601752 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43136_solr
   [junit4]   2> 2601754 INFO  (zkCallback-8202-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601754 INFO  (zkCallback-8235-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601754 INFO  (zkCallback-8224-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601754 INFO  (zkCallback-8240-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601754 INFO  (zkCallback-8222-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601754 INFO  (zkCallback-8211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601754 INFO  (zkCallback-8197-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601775 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2601836 INFO  (zkCallback-8264-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601836 INFO  (zkCallback-8248-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601842 INFO  (zkCallback-8253-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601843 INFO  (zkCallback-8261-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (6)
   [junit4]   2> 2601847 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36836.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601849 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44645.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601850 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44645.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601851 INFO  (jetty-launcher-8171-thread-2) [n:127.0.0.1:44645_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001/node2/.
   [junit4]   2> 2601851 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_32952.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601860 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2601896 INFO  (jetty-launcher-8171-thread-8) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43098/solr
   [junit4]   2> 2601903 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_32952.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601904 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_32952.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601908 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2601913 INFO  (zkConnectionManagerCallback-8275-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601915 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 2601918 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2601919 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2601926 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36836.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601926 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36836.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601925 INFO  (jetty-launcher-8171-thread-10) [n:127.0.0.1:32952_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001/node10/.
   [junit4]   2> 2601934 INFO  (jetty-launcher-8171-thread-6) [n:127.0.0.1:36836_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001/node6/.
   [junit4]   2> 2601947 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43136.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601953 INFO  (zkConnectionManagerCallback-8279-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601962 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43136.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601962 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43136.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2601964 INFO  (jetty-launcher-8171-thread-5) [n:127.0.0.1:43136_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001/node5/.
   [junit4]   2> 2601968 INFO  (jetty-launcher-8171-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43098/solr
   [junit4]   2> 2601970 INFO  (jetty-launcher-8171-thread-7) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43098/solr
   [junit4]   2> 2601980 INFO  (zkConnectionManagerCallback-8284-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601981 INFO  (zkConnectionManagerCallback-8287-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601981 INFO  (zkConnectionManagerCallback-8289-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2601994 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 2602003 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:36980_solr as DOWN
   [junit4]   2> 2602004 INFO  (zkConnectionManagerCallback-8291-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2602005 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2602005 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36980_solr
   [junit4]   2> 2602011 INFO  (zkConnectionManagerCallback-8295-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2602012 INFO  (zkCallback-8264-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602012 INFO  (zkCallback-8240-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602012 INFO  (zkCallback-8202-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602012 INFO  (zkCallback-8224-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602013 INFO  (zkCallback-8197-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602013 INFO  (zkCallback-8211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602013 INFO  (zkCallback-8222-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602013 INFO  (zkCallback-8235-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602020 INFO  (zkCallback-8248-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602020 INFO  (zkCallback-8261-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602021 INFO  (zkCallback-8253-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602024 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 2602025 INFO  (zkCallback-8288-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602027 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
   [junit4]   2> 2602041 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:45731_solr as DOWN
   [junit4]   2> 2602044 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2602045 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45731_solr
   [junit4]   2> 2602047 INFO  (zkCallback-8264-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (zkCallback-8253-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (zkCallback-8224-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (zkCallback-8202-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (zkCallback-8235-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (zkCallback-8197-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (zkCallback-8222-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (zkCallback-8211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (zkCallback-8261-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (zkCallback-8240-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (zkCallback-8248-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
   [junit4]   2> 2602048 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:36671_solr as DOWN
   [junit4]   2> 2602049 INFO  (zkCallback-8274-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (7)
   [junit4]   2> 2602049 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2602050 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36671_solr
   [junit4]   2> 2602051 INFO  (zkCallback-8264-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602052 INFO  (zkCallback-8202-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602052 INFO  (zkCallback-8224-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602052 INFO  (zkCallback-8197-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602052 INFO  (zkCallback-8248-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602052 INFO  (zkCallback-8235-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602052 INFO  (zkCallback-8211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602052 INFO  (zkCallback-8261-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602052 INFO  (zkCallback-8253-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602053 INFO  (zkCallback-8240-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602053 INFO  (zkCallback-8222-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
   [junit4]   2> 2602074 INFO  (zkCallback-8288-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 2602074 INFO  (zkCallback-8294-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 2602074 INFO  (zkCallback-8290-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 2602075 INFO  (zkCallback-8274-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
   [junit4]   2> 2602111 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2602125 INFO  (zkConnectionManagerCallback-8309-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2602126 INFO  (jetty-launcher-8171-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43098/solr
   [junit4]   2> 2602128 INFO  (zkConnectionManagerCallback-8315-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2602128 INFO  (zkConnectionManagerCallback-8304-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2602129 INFO  (zkConnectionManagerCallback-8318-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2602129 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (9)
   [junit4]   2> 2602130 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (9)
   [junit4]   2> 2602130 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (9)
   [junit4]   2> 2602132 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2602133 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2602134 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2602134 INFO  (zkConnectionManagerCallback-8321-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2602140 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2602140 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2602145 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2602145 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (9)
   [junit4]   2> 2602164 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:42771_solr as DOWN
   [junit4]   2> 2602172 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2602172 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42771_solr
   [junit4]   2> 2602175 INFO  (zkCallback-8253-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602175 INFO  (zkCallback-8294-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602175 INFO  (zkCallback-8288-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602175 INFO  (zkCallback-8235-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602175 INFO  (zkCallback-8290-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602175 INFO  (zkCallback-8261-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602175 INFO  (zkCallback-8264-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602175 INFO  (zkCallback-8211-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602175 INFO  (zkCallback-8248-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602175 INFO  (zkCallback-8197-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602176 INFO  (zkCallback-8240-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602176 INFO  (zkCallback-8224-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602176 INFO  (zkCallback-8202-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602176 INFO  (zkCallback-8222-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602177 INFO  (zkCallback-8314-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602177 INFO  (zkCallback-8320-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602177 INFO  (zkCallback-8274-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602179 INFO  (zkCallback-8308-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602184 INFO  (zkCallback-8303-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
   [junit4]   2> 2602256 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36980.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602276 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45731.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602284 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36671.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602302 INFO  (zkConnectionManagerCallback-8330-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2602305 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (10)
   [junit4]   2> 2602308 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2602325 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45731.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602325 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45731.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602326 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36980.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602326 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36980.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602328 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2602338 INFO  (jetty-launcher-8171-thread-7) [n:127.0.0.1:45731_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001/node7/.
   [junit4]   2> 2602338 INFO  (jetty-launcher-8171-thread-8) [n:127.0.0.1:36980_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001/node8/.
   [junit4]   2> 2602351 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36671.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602351 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36671.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602355 INFO  (jetty-launcher-8171-thread-1) [n:127.0.0.1:36671_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001/node1/.
   [junit4]   2> 2602391 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42771.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602414 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42771.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602414 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42771.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a19d150
   [junit4]   2> 2602417 INFO  (jetty-launcher-8171-thread-3) [n:127.0.0.1:42771_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_99829954D323FA78-001/tempDir-001/node3/.
   [junit4]   2> 2603325 INFO  (zkConnectionManagerCallback-8336-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2603327 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (10)
   [junit4]   2> 2603328 INFO  (SUITE-TestStressCloudBlindAtomicUpdates-seed#[99829954D323FA78]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43098/solr ready
   [junit4]   2> 2603984 INFO  (qtp1830078810-92269) [n:127.0.0.1:43002_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params property.config=solrconfig-tlog.xml&property.schema=schema-minimal-atomic-stress.xml&collection.configName=org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates_config-set&name=test_col&nrtReplicas=2&action=CREATE&numShards=5&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2603996 INFO  (OverseerThreadFactory-7832-thread-1-processing-n:127.0.0.1:43002_solr) [n:127.0.0.1:43002_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection test_col
   [junit4]   2> 2604107 INFO  (qtp1830078810-92271) [n:127.0.0.1:43002_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604108 INFO  (qtp1830078810-92272) [n:127.0.0.1:43002_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604109 INFO  (qtp1830078810-92268) [n:127.0.0.1:43002_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604111 INFO  (qtp1830078810-92270) [n:127.0.0.1:43002_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 2604111 INFO  (qtp1830078810-92271) [n:127.0.0.1:43002_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604113 INFO  (qtp184121485-92318) [n:127.0.0.1:32952_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604114 INFO  (qtp184121485-92320) [n:127.0.0.1:32952_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604115 INFO  (qtp184121485-92319) [n:127.0.0.1:32952_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604116 INFO  (qtp184121485-92321) [n:127.0.0.1:32952_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604117 INFO  (qtp184121485-92256) [n:127.0.0.1:32952_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604119 INFO  (qtp968952409-92279) [n:127.0.0.1:36980_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604120 INFO  (qtp968952409-92306) [n:127.0.0.1:36980_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604120 INFO  (qtp968952409-92307) [n:127.0.0.1:36980_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604121 INFO  (qtp968952409-92308) [n:127.0.0.1:36980_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604122 INFO  (qtp968952409-92277) [n:127.0.0.1:36980_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604124 INFO  (qtp1611574244-92261) [n:127.0.0.1:38131_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604125 INFO  (qtp1611574244-92259) [n:127.0.0.1:38131_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604126 INFO  (qtp1611574244-92262) [n:127.0.0.1:38131_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604127 INFO  (qtp1611574244-92263) [n:127.0.0.1:38131_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604134 INFO  (qtp1611574244-92260) [n:127.0.0.1:38131_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604136 INFO  (qtp178141026-92284) [n:127.0.0.1:45731_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604137 INFO  (qtp178141026-92278) [n:127.0.0.1:45731_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604138 INFO  (qtp178141026-92285) [n:127.0.0.1:45731_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604139 INFO  (qtp178141026-92287) [n:127.0.0.1:45731_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604141 INFO  (qtp178141026-92283) [n:127.0.0.1:45731_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604147 INFO  (qtp351736604-92291) [n:127.0.0.1:36836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604148 INFO  (qtp351736604-92292) [n:127.0.0.1:36836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604149 INFO  (qtp351736604-92293) [n:127.0.0.1:36836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604150 INFO  (qtp351736604-92294) [n:127.0.0.1:36836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604151 INFO  (qtp351736604-92290) [n:127.0.0.1:36836_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604158 INFO  (qtp1656368709-92247) [n:127.0.0.1:44645_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604159 INFO  (qtp1656368709-92246) [n:127.0.0.1:44645_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604160 INFO  (qtp1656368709-92248) [n:127.0.0.1:44645_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604161 INFO  (qtp1656368709-92249) [n:127.0.0.1:44645_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604162 INFO  (qtp1656368709-92245) [n:127.0.0.1:44645_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2604165 INFO  (qtp2022194171-92300) [n:127.0.

[...truncated too long message...]

.7/checkout/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/bcprov-jdk15on-1.60.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/byte-buddy-1.9.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/mockito-core-2.23.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/objenesis-2.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-7.7.1-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/backward-codecs/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/contrib/analysis-extras/lib/icu4j-62.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_191/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.7.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/temp/junit4-J1-20190214_001757_6509144916926725075701.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/temp/junit4-J1-20190214_001757_6502363150548669828520.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1563)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:123)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:997)
   [junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:994)
   [junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4] at java.lang.Thread.run(Thread.java:748)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:651: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:577: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build.xml:267: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/common-build.xml:558: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/common-build.xml:1572: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/common-build.xml:1099: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 252 minutes 51 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-NightlyTests-7.7 #9 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 11 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/11/

3 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:40181

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:40181
        at __randomizedtesting.SeedInfo.seed([C575C4BD9A19594B:4D21FB6734E534B3]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:338)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1073)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 43 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest:     1) Thread[id=12085, name=zkConnectionManagerCallback-3289-thread-1, state=WAITING, group=TGRP-AutoscalingHistoryHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest:
   1) Thread[id=12085, name=zkConnectionManagerCallback-3289-thread-1, state=WAITING, group=TGRP-AutoscalingHistoryHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([C575C4BD9A19594B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=12085, name=zkConnectionManagerCallback-3289-thread-1, state=WAITING, group=TGRP-AutoscalingHistoryHandlerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=12085, name=zkConnectionManagerCallback-3289-thread-1, state=WAITING, group=TGRP-AutoscalingHistoryHandlerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([C575C4BD9A19594B]:0)




Build Log:
[...truncated 13239 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.HttpPartitionTest_C575C4BD9A19594B-001/init-core-data-001
   [junit4]   2> 8359 INFO  (SUITE-HttpPartitionTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 8613 INFO  (SUITE-HttpPartitionTest-seed#[C575C4BD9A19594B]-worker) [    ] o.e.j.u.log Logging initialized @8615ms to org.eclipse.jetty.util.log.Slf4jLog
   [junit4]   2> 8676 INFO  (SUITE-HttpPartitionTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 8723 INFO  (SUITE-HttpPartitionTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 8729 INFO  (SUITE-HttpPartitionTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 8904 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 8918 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 8920 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 9545 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer start zk server on port:45169
   [junit4]   2> 9596 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:45169
   [junit4]   2> 10112 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 45169
   [junit4]   2> 11742 INFO  (zkConnectionManagerCallback-3-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 12323 INFO  (zkConnectionManagerCallback-5-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 12615 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 18764 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 18801 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 18840 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 18911 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 18927 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 18929 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 18984 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 19020 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 19053 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 19055 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 19074 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 24631 WARN  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 25904 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 25904 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 25962 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 26438 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 26438 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 26459 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 26622 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@a57f259{/,null,AVAILABLE}
   [junit4]   2> 26723 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1e04c92e{HTTP/1.1,[http/1.1]}{127.0.0.1:37672}
   [junit4]   2> 26723 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.e.j.s.Server Started @26783ms
   [junit4]   2> 26723 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.HttpPartitionTest_C575C4BD9A19594B-001/tempDir-001/control/data, hostContext=/, hostPort=40181, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.HttpPartitionTest_C575C4BD9A19594B-001/control-001/cores}
   [junit4]   2> 27458 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.u.c.SSLCredentialProviderFactory Processing SSL Credential Provider chain: env;sysprop
   [junit4]   2> 28077 ERROR (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 28077 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 28077 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.1
   [junit4]   2> 28095 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 28111 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 28112 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-15T05:26:23.337Z
   [junit4]   2> 28404 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 28504 INFO  (zkConnectionManagerCallback-7-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 28520 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 28645 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.HttpPartitionTest_C575C4BD9A19594B-001/control-001/solr.xml
   [junit4]   2> 29378 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 29379 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 29668 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@356cfe47, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 33232 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 36569 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45169/solr
   [junit4]   2> 37353 INFO  (zkConnectionManagerCallback-11-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 37572 INFO  (zkConnectionManagerCallback-13-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 40803 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [n:127.0.0.1:40181_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40181_
   [junit4]   2> 40874 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [n:127.0.0.1:40181_    ] o.a.s.c.Overseer Overseer (id=74190513376854020-127.0.0.1:40181_-n_0000000000) starting
   [junit4]   2> 42004 INFO  (zkConnectionManagerCallback-20-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 42099 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [n:127.0.0.1:40181_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45169/solr ready
   [junit4]   2> 43896 INFO  (OverseerStateUpdate-74190513376854020-127.0.0.1:40181_-n_0000000000) [n:127.0.0.1:40181_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:40181_
   [junit4]   2> 43934 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [n:127.0.0.1:40181_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40181_
   [junit4]   2> 45066 INFO  (OverseerStateUpdate-74190513376854020-127.0.0.1:40181_-n_0000000000) [n:127.0.0.1:40181_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 45406 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 48942 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [n:127.0.0.1:40181_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 50579 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [n:127.0.0.1:40181_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@356cfe47
   [junit4]   2> 50613 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [n:127.0.0.1:40181_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@356cfe47
   [junit4]   2> 50615 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [n:127.0.0.1:40181_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@356cfe47
   [junit4]   2> 51014 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [n:127.0.0.1:40181_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.HttpPartitionTest_C575C4BD9A19594B-001/control-001/cores
   [junit4]   2> 52581 INFO  (zkConnectionManagerCallback-26-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 52802 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 52803 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45169/solr ready
   [junit4]   2> 53879 INFO  (SocketProxy-Acceptor-40181) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=39406,localport=40181], receiveBufferSize:531000
   [junit4]   2> 53988 INFO  (SocketProxy-Acceptor-40181) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37672,localport=48884], receiveBufferSize=530904
   [junit4]   2> 56018 INFO  (qtp180787263-30) [n:127.0.0.1:40181_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:40181_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 56326 INFO  (OverseerThreadFactory-10-thread-1-processing-n:127.0.0.1:40181_) [n:127.0.0.1:40181_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 57273 INFO  (SocketProxy-Acceptor-40181) [    ] o.a.s.c.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=39420,localport=40181], receiveBufferSize:531000
   [junit4]   2> 57289 INFO  (SocketProxy-Acceptor-40181) [    ] o.a.s.c.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37672,localport=48896], receiveBufferSize=530904
   [junit4]   2> 57510 INFO  (qtp180787263-32) [n:127.0.0.1:40181_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=162
   [junit4]   2> 58151 INFO  (qtp180787263-33) [n:127.0.0.1:40181_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=39
   [junit4]   2> 58187 INFO  (qtp180787263-29) [n:127.0.0.1:40181_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 58204 INFO  (qtp180787263-31) [n:127.0.0.1:40181_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 58223 INFO  (qtp180787263-32) [n:127.0.0.1:40181_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 60040 INFO  (qtp180787263-33) [n:127.0.0.1:40181_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 60061 INFO  (qtp180787263-33) [n:127.0.0.1:40181_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 62173 INFO  (qtp180787263-33) [n:127.0.0.1:40181_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.1
   [junit4]   2> 64357 INFO  (qtp180787263-33) [n:127.0.0.1:40181_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 65227 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=86551503
   [junit4]   2> 65227 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:40181_
   [junit4]   2> 65298 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.c.CoreContainer Publish this node as DOWN...
   [junit4]   2> 65298 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.c.ZkController Publish node=127.0.0.1:40181_ as DOWN
   [junit4]   2> 65841 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 65892 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@634a47fb: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@7cb378d7
   [junit4]   2> 66729 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 66858 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@159a2cc5: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@ec5daf2
   [junit4]   2> 66877 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 66878 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@61ea0b37: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@45bcd434
   [junit4]   2> 66984 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 67038 INFO  (closeThreadPool-31-thread-2) [    ] o.a.s.c.Overseer Overseer (id=74190513376854020-127.0.0.1:40181_-n_0000000000) closing
   [junit4]   2> 67077 INFO  (OverseerStateUpdate-74190513376854020-127.0.0.1:40181_-n_0000000000) [n:127.0.0.1:40181_    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:40181_
   [junit4]   2> 67240 ERROR (OverseerThreadFactory-10-thread-1-processing-n:127.0.0.1:40181_) [n:127.0.0.1:40181_    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard: http://127.0.0.1:40181
   [junit4]   2> org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:40181
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) ~[java/:?]
   [junit4]   2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260) ~[java/:?]
   [junit4]   2> at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:173) ~[java/:?]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_191]
   [junit4]   2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) ~[metrics-core-3.2.6.jar:3.2.6]
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
   [junit4]   2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
   [junit4]   2> Caused by: java.net.SocketException: Socket closed
   [junit4]   2> at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_191]
   [junit4]   2> at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_191]
   [junit4]   2> at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.10.jar:4.4.10]
   [junit4]   2> at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.10.jar:4.4.10]
   [junit4]   2> at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[httpcore-4.4.10.jar:4.4.10]
   [junit4]   2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.6.jar:4.5.6]
   [junit4]   2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.6.jar:4.5.6]
   [junit4]   2> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.10.jar:4.4.10]
   [junit4]   2> at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.10.jar:4.4.10]
   [junit4]   2> at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[httpclient-4.5.6.jar:4.5.6]
   [junit4]   2> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.10.jar:4.4.10]
   [junit4]   2> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.10.jar:4.4.10]
   [junit4]   2> at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:120) ~[java/:?]
   [junit4]   2> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.6.jar:4.5.6]
   [junit4]   2> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.6.jar:4.5.6]
   [junit4]   2> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.6.jar:4.5.6]
   [junit4]   2> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.6.jar:4.5.6]
   [junit4]   2> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.6.jar:4.5.6]
   [junit4]   2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.6.jar:4.5.6]
   [junit4]   2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar:4.5.6]
   [junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542) ~[java/:?]
   [junit4]   2> ... 12 more
   [junit4]   2> 67479 ERROR (OverseerThreadFactory-10-thread-1-processing-n:127.0.0.1:40181_) [n:127.0.0.1:40181_    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Cleaning up collection [control_collection].
   [junit4]   2> 67552 ERROR (OverseerThreadFactory-10-thread-1-processing-n:127.0.0.1:40181_) [n:127.0.0.1:40181_    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: control_collection operation: create failed:org.apache.solr.common.SolrException: Could not load collection from ZK: control_collection
   [junit4]   2> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1363)
   [junit4]   2> at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:745)
   [junit4]   2> at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:148)
   [junit4]   2> at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:131)
   [junit4]   2> at org.apache.solr.cloud.api.collections.DeleteCollectionCmd.checkNotColocatedWith(DeleteCollectionCmd.java:193)
   [junit4]   2> at org.apache.solr.cloud.api.collections.DeleteCollectionCmd.call(DeleteCollectionCmd.java:75)
   [junit4]   2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.cleanupCollection(OverseerCollectionMessageHandler.java:710)
   [junit4]   2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:290)
   [junit4]   2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:292)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:496)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/control_collection/state.json
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358)
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358)
   [junit4]   2> at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1375)
   [junit4]   2> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1361)
   [junit4]   2> ... 13 more
   [junit4]   2>
   [junit4]   2> 67915 ERROR (OverseerThreadFactory-10-thread-1-processing-n:127.0.0.1:40181_) [n:127.0.0.1:40181_    ] o.a.s.c.OverseerTaskProcessor :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qnr-0000000000
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334)
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskQueue.remove(OverseerTaskQueue.java:96)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.markTaskComplete(OverseerTaskProcessor.java:564)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:516)
   [junit4]   2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 67915 WARN  (OverseerThreadFactory-10-thread-1-processing-n:127.0.0.1:40181_) [n:127.0.0.1:40181_    ] o.a.s.c.OverseerTaskProcessor Resetting task: /overseer/collection-queue-work/qn-0000000000, requestid: null, taskKey: control_collection
   [junit4]   2> 67916 INFO  (OverseerAutoScalingTriggerThread-74190513376854020-127.0.0.1:40181_-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 67931 INFO  (closeThreadPool-28-thread-2) [    ] o.a.s.c.Overseer Overseer (id=74190513376854020-127.0.0.1:40181_-n_0000000000) closing
   [junit4]   2> 67931 INFO  (closeThreadPool-31-thread-1) [    ] o.a.s.c.Overseer Overseer (id=74190513376854020-127.0.0.1:40181_-n_0000000000) closing
   [junit4]   2> 68077 INFO  (closeThreadPool-28-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1e04c92e{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 68135 INFO  (closeThreadPool-28-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@a57f259{/,null,UNAVAILABLE}
   [junit4]   2> 68170 INFO  (closeThreadPool-28-thread-2) [    ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 71288 WARN  (qtp180787263-33) [n:127.0.0.1:40181_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.LowerCaseTokenizerFactory]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 72727 WARN  (qtp180787263-33) [n:127.0.0.1:40181_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.WordDelimiterFilterFactory]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 73769 WARN  (qtp180787263-33) [n:127.0.0.1:40181_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.GeoHashField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 73788 WARN  (qtp180787263-33) [n:127.0.0.1:40181_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.LatLonType]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 73932 WARN  (qtp180787263-33) [n:127.0.0.1:40181_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.CurrencyField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 74798 ERROR (qtp180787263-33) [n:127.0.0.1:40181_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer null:org.apache.solr.common.SolrException: Error creating shard term node in Zookeeper for collection: control_collection
   [junit4]   2> at org.apache.solr.cloud.ZkShardTerms.ensureTermNodeExist(ZkShardTerms.java:332)
   [junit4]   2> at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:90)
   [junit4]   2> at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:45)
   [junit4]   2> at org.apache.solr.cloud.ZkCollectionTerms.remove(ZkCollectionTerms.java:52)
   [junit4]   2> at org.apache.solr.cloud.ZkController.unregister(ZkController.java:1636)
   [junit4]   2> at org.apache.solr.cloud.ZkController.unregister(ZkController.java:1630)
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1101)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:502)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$makePath$8(SolrZkClient.java:545)
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:544)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:488)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:475)
   [junit4]   2> at org.apache.solr.cloud.ZkShardTerms.ensureTermNodeExist(ZkShardTerms.java:322)
   [junit4]   2> ... 41 more
   [junit4]   2>
   [junit4]   2> 74799 ERROR (qtp180787263-33) [n:127.0.0.1:40181_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'control_collection_shard1_replica_n1': Unable to create core [control_collection_shard1_replica_n1] Caused by: KeeperErrorCode = Session expired for /configs/conf1/enumsConfig.xml
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1127)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
   [junit4]   2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:502)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unable to create core [control_collection_shard1_replica_n1]
   [junit4]   2> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1199)
   [junit4]   2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1088)
   [junit4]   2> ... 35 more
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Could not load conf for core control_collection_shard1_replica_n1: Can't load schema schema.xml: Plugin Initializing failure for [schema.xml] fieldType
   [junit4]   2> at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:97)
   [junit4]   2> at org.apache.solr.core.CoreContainer.getConfigSet(CoreContainer.java:1227)
   [junit4]   2> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1174)
   [junit4]   2> ... 36 more
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Can't load schema schema.xml: Plugin Initializing failure for [schema.xml] fieldType
   [junit4]   2> at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:590)
   [junit4]   2> at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:180)
   [junit4]   2> at org.apache.solr.schema.IndexSchemaFactory.create(IndexSchemaFactory.java:56)
   [junit4]   2> at org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
   [junit4]   2> at org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:119)
   [junit4]   2> at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:92)
   [junit4]   2> ... 38 more
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Plugin Initializing failure for [schema.xml] fieldType
   [junit4]   2> at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:194)
   [junit4]   2> at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:481)
   [junit4]   2> ... 43 more
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: severityType: Error while parsing enums config.
   [junit4]   2> at org.apache.solr.schema.AbstractEnumField$EnumMapping.<init>(AbstractEnumField.java:150)
   [junit4]   2> at org.apache.solr.schema.AbstractEnumField.init(AbstractEnumField.java:59)
   [junit4]   2> at org.apache.solr.schema.FieldType.setArgs(FieldType.java:193)
   [junit4]   2> at org.apache.solr.schema.FieldTypePluginLoader.init(FieldTypePluginLoader.java:165)
   [junit4]   2> at org.apache.solr.schema.FieldTypePluginLoader.init(FieldTypePluginLoader.java:53)
   [junit4]   2> at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:191)
   [junit4]   2> ... 44 more
   [junit4]   2> Caused by: java.io.IOException: We re-tried 10 times but was still unable to fetch resource=enumsConfig.xml from ZK
   [junit4]   2> at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:118)
   [junit4]   2> at org.apache.solr.util.SafeXMLParsing.parseConfigXML(SafeXMLParsing.java:56)
   [junit4]   2> at org.apache.solr.schema.AbstractEnumField$EnumMapping.<init>(AbstractEnumField.java:112)
   [junit4]   2> ... 49 more
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /configs/conf1/enumsConfig.xml
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334)
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334)
   [junit4]   2> at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:1120)
   [junit4]   2> at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
   [junit4]   2> ... 51 more
   [junit4]   2>
   [junit4]   2> 74806 INFO  (qtp180787263-33) [n:127.0.0.1:40181_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=400 QTime=14882
   [junit4]   2> 83196 ERROR (qtp180787263-30) [n:127.0.0.1:40181_    ] o.a.s.h.RequestHandlerBase java.lang.InterruptedException
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2173)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskQueue$LatchWatcher.await(OverseerTaskQueue.java:158)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskQueue.offer(OverseerTaskQueue.java:208)
   [junit4]   2> at org.apache.solr.handler.admin.CollectionsHandler.sendToOCPQueue(CollectionsHandler.java:363)
   [junit4]   2> at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:273)
   [junit4]   2> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:247)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:502)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 83197 INFO  (qtp180787263-30) [n:127.0.0.1:40181_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:40181_&wt=javabin&version=2} status=500 QTime=27667
   [junit4]   2> 83197 ERROR (qtp180787263-30) [n:127.0.0.1:40181_    ] o.a.s.s.HttpSolrCall null:java.lang.InterruptedException
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2173)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskQueue$LatchWatcher.await(OverseerTaskQueue.java:158)
   [junit4]   2> at org.apache.solr.cloud.OverseerTaskQueue.offer(OverseerTaskQueue.java:208)
   [junit4]   2> at org.apache.solr.handler.admin.CollectionsHandler.sendToOCPQueue(CollectionsHandler.java:363)
   [junit4]   2> at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:273)
   [junit4]   2> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:247)
   [junit4]   2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
   [junit4]   2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
   [junit4]   2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:158)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
   [junit4]   2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
   [junit4]   2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
   [junit4]   2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
   [junit4]   2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
   [junit4]   2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
   [junit4]   2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
   [junit4]   2> at org.eclipse.jetty.server.Server.handle(Server.java:502)
   [junit4]   2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
   [junit4]   2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
   [junit4]   2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
   [junit4]   2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
   [junit4]   2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>
   [junit4]   2> 83281 WARN  (closeThreadPool-28-thread-2) [    ] o.a.s.c.s.c.SocketProxy Closing 2 connections to: http://127.0.0.1:40181/, target: http://127.0.0.1:37672/
   [junit4]   2> 83282 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
   [junit4]   2> 83366 WARN  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 3 /solr/aliases.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 3 /solr/clusterstate.json
   [junit4]   2> 3 /solr/clusterprops.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 3 /solr/live_nodes
   [junit4]   2> 3 /solr/collections
   [junit4]   2>
   [junit4]   2> 83613 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:45169
   [junit4]   2> 83613 INFO  (TEST-HttpPartitionTest.test-seed#[C575C4BD9A19594B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 45169
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HttpPartitionTest -Dtests.method=test -Dtests.seed=C575C4BD9A19594B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/test-data/enwiki.random.lines.txt -Dtests.locale=et -Dtests.timezone=Africa/Timbuktu -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   75.0s J2 | HttpPartitionTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:40181
   [junit4]    > at __randomizedtesting.SeedInfo.seed([C575C4BD9A19594B:4D21FB6734E534B3]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
   [junit4]    > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:338)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1073)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]    > at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
   [junit4]    > at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
   [junit4]    > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
   [junit4]    > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   [junit4]    > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   [junit4]    > at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
   [junit4]    > at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
   [junit4]    > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   [junit4]    > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   [junit4]    > at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
   [junit4]    > at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   [junit4]    > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
   [junit4]    > at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]    > at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   [junit4]    > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   [junit4]    > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
   [junit4]    > ... 43 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.HttpPartitionTest_C575C4BD9A19594B-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1784, maxMBSortInHeap=7.0287502220895846, sim=RandomSimilarity(queryNorm=true): {}, locale=et, timezone=Africa/Timbuktu
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=120992568,total=262668288
   [junit4]   2> NOTE: All tests run in this JVM: [HttpPartitionTest]
   [junit4] Completed [2/838 (1!)] on J2 in 81.62s, 1 test, 1 error <<< FAILURES!

[...truncated 607 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_C575C4BD9A19594B-001/init-core-data-001
   [junit4]   2> 1796990 WARN  (SUITE-AutoscalingHistoryHandlerTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=14 numCloses=14
   [junit4]   2> 1796990 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1796992 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1796992 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1796992 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_C575C4BD9A19594B-001/tempDir-001
   [junit4]   2> 1796992 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1796993 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1796993 INFO  (ZkTestServer Run Thread) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1797093 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:46122
   [junit4]   2> 1797093 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:46122
   [junit4]   2> 1797093 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 46122
   [junit4]   2> 1797096 INFO  (zkConnectionManagerCallback-3171-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797099 INFO  (zkConnectionManagerCallback-3173-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797103 INFO  (zkConnectionManagerCallback-3175-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797105 WARN  (jetty-launcher-3176-thread-1) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1797105 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1797105 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1797105 INFO  (jetty-launcher-3176-thread-1) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1797106 WARN  (jetty-launcher-3176-thread-2) [    ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
   [junit4]   2> 1797106 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
   [junit4]   2> 1797106 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 1797106 INFO  (jetty-launcher-3176-thread-2) [    ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
   [junit4]   2> 1797107 INFO  (jetty-launcher-3176-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1797107 INFO  (jetty-launcher-3176-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1797107 INFO  (jetty-launcher-3176-thread-1) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1797107 INFO  (jetty-launcher-3176-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1797107 INFO  (jetty-launcher-3176-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1797107 INFO  (jetty-launcher-3176-thread-2) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1797108 INFO  (jetty-launcher-3176-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@22fffa3{/solr,null,AVAILABLE}
   [junit4]   2> 1797108 INFO  (jetty-launcher-3176-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3e4e9567{/solr,null,AVAILABLE}
   [junit4]   2> 1797117 INFO  (jetty-launcher-3176-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@449e1346{HTTP/1.1,[http/1.1]}{127.0.0.1:39369}
   [junit4]   2> 1797117 INFO  (jetty-launcher-3176-thread-1) [    ] o.e.j.s.Server Started @1797177ms
   [junit4]   2> 1797118 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39369}
   [junit4]   2> 1797118 ERROR (jetty-launcher-3176-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1797118 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1797118 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.1
   [junit4]   2> 1797118 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1797118 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1797118 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-15T05:55:52.343Z
   [junit4]   2> 1797119 INFO  (jetty-launcher-3176-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@57207659{HTTP/1.1,[http/1.1]}{127.0.0.1:35244}
   [junit4]   2> 1797119 INFO  (jetty-launcher-3176-thread-2) [    ] o.e.j.s.Server Started @1797179ms
   [junit4]   2> 1797119 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35244}
   [junit4]   2> 1797119 ERROR (jetty-launcher-3176-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1797119 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1797119 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.7.1
   [junit4]   2> 1797119 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1797119 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1797119 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2019-02-15T05:55:52.344Z
   [junit4]   2> 1797120 INFO  (zkConnectionManagerCallback-3178-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797120 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1797146 INFO  (zkConnectionManagerCallback-3180-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797146 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1797255 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1797282 INFO  (jetty-launcher-3176-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46122/solr
   [junit4]   2> 1797298 INFO  (zkConnectionManagerCallback-3184-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797300 INFO  (zkConnectionManagerCallback-3186-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797411 INFO  (jetty-launcher-3176-thread-2) [n:127.0.0.1:35244_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35244_solr
   [junit4]   2> 1797412 INFO  (jetty-launcher-3176-thread-2) [n:127.0.0.1:35244_solr    ] o.a.s.c.Overseer Overseer (id=74190630481559558-127.0.0.1:35244_solr-n_0000000000) starting
   [junit4]   2> 1797437 INFO  (zkConnectionManagerCallback-3193-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797451 INFO  (jetty-launcher-3176-thread-2) [n:127.0.0.1:35244_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46122/solr ready
   [junit4]   2> 1797452 INFO  (OverseerStateUpdate-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [n:127.0.0.1:35244_solr    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:35244_solr
   [junit4]   2> 1797453 INFO  (jetty-launcher-3176-thread-2) [n:127.0.0.1:35244_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35244_solr
   [junit4]   2> 1797466 INFO  (OverseerStateUpdate-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [n:127.0.0.1:35244_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1797466 DEBUG (OverseerAutoScalingTriggerThread-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 1797479 DEBUG (OverseerAutoScalingTriggerThread-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1797479 DEBUG (OverseerAutoScalingTriggerThread-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 1797479 DEBUG (OverseerAutoScalingTriggerThread-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1797486 INFO  (zkCallback-3192-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1797497 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 1797501 INFO  (jetty-launcher-3176-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46122/solr
   [junit4]   2> 1797501 DEBUG (OverseerAutoScalingTriggerThread-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:35244_solr]
   [junit4]   2> 1797507 INFO  (jetty-launcher-3176-thread-2) [n:127.0.0.1:35244_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1797510 DEBUG (ScheduledTrigger-3339-thread-2) [    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 1797526 DEBUG (OverseerAutoScalingTriggerThread-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
   [junit4]   2> 1797526 DEBUG (OverseerAutoScalingTriggerThread-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1797528 INFO  (zkConnectionManagerCallback-3197-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797538 INFO  (jetty-launcher-3176-thread-2) [n:127.0.0.1:35244_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35244.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@356cfe47
   [junit4]   2> 1797542 INFO  (zkConnectionManagerCallback-3200-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797555 INFO  (jetty-launcher-3176-thread-2) [n:127.0.0.1:35244_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35244.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@356cfe47
   [junit4]   2> 1797555 INFO  (jetty-launcher-3176-thread-2) [n:127.0.0.1:35244_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35244.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@356cfe47
   [junit4]   2> 1797556 INFO  (jetty-launcher-3176-thread-2) [n:127.0.0.1:35244_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_C575C4BD9A19594B-001/tempDir-001/node2/.
   [junit4]   2> 1797558 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1797572 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:39369_solr as DOWN
   [junit4]   2> 1797573 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1797573 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39369_solr
   [junit4]   2> 1797573 DEBUG (OverseerStateUpdate-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [n:127.0.0.1:35244_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:39369_solr"} current state version: 0
   [junit4]   2> 1797573 DEBUG (OverseerStateUpdate-74190630481559558-127.0.0.1:35244_solr-n_0000000000) [n:127.0.0.1:35244_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:39369_solr
   [junit4]   2> 1797574 INFO  (zkCallback-3185-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1797574 INFO  (zkCallback-3192-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1797574 INFO  (zkCallback-3199-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1797665 INFO  (zkConnectionManagerCallback-3207-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797666 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1797667 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46122/solr ready
   [junit4]   2> 1797667 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1797684 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39369.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@356cfe47
   [junit4]   2> 1797692 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39369.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@356cfe47
   [junit4]   2> 1797693 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39369.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@356cfe47
   [junit4]   2> 1797694 INFO  (jetty-launcher-3176-thread-1) [n:127.0.0.1:39369_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_C575C4BD9A19594B-001/tempDir-001/node1/.
   [junit4]   2> 1797727 INFO  (zkConnectionManagerCallback-3213-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1797729 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[C575C4BD9A19594B]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes f

[...truncated too long message...]

esource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null46506914
     [copy] Copying 242 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null46506914
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/null46506914

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 112ms :: artifacts dl 11ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 415 minutes 13 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2847)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2726)
        at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2707)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
                at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1835)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3088)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath.act(FilePath.java:1074)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2705)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]