[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 12 - Still unstable

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 12 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/12/

1 tests failed.
FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
        at __randomizedtesting.SeedInfo.seed([6B327E92B5F4887:8EE7183385A3257F]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:317)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12041 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/init-core-data-001
   [junit4]   2> 908822 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=8 numCloses=8
   [junit4]   2> 908822 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 908823 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 908823 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 908853 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 908860 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 908863 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 908872 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_55141_hdfs____.m8fjig/webapp
   [junit4]   2> 909246 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55141
   [junit4]   2> 909307 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 909308 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 909320 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_59155_datanode____.akm2p8/webapp
   [junit4]   2> 909724 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59155
   [junit4]   2> 909776 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 909777 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 909787 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_56195_datanode____.ioq0r1/webapp
   [junit4]   2> 909837 INFO  (IPC Server handler 3 on 49935) [    ] BlockStateChange BLOCK* processReport: from storage DS-c136754a-c95c-4594-8701-6a06002f4aeb node DatanodeRegistration(127.0.0.1:44594, datanodeUuid=531c9885-8f86-4c31-942e-ce24f0d0d7c2, infoPort=47542, infoSecurePort=0, ipcPort=60652, storageInfo=lv=-56;cid=testClusterID;nsid=564158542;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 909837 INFO  (IPC Server handler 3 on 49935) [    ] BlockStateChange BLOCK* processReport: from storage DS-44f2195a-d4d5-4b95-806f-bb32c87db2ba node DatanodeRegistration(127.0.0.1:44594, datanodeUuid=531c9885-8f86-4c31-942e-ce24f0d0d7c2, infoPort=47542, infoSecurePort=0, ipcPort=60652, storageInfo=lv=-56;cid=testClusterID;nsid=564158542;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 910199 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56195
   [junit4]   2> 910284 INFO  (IPC Server handler 8 on 49935) [    ] BlockStateChange BLOCK* processReport: from storage DS-b0216fe9-73f7-4d12-8c5e-d4a4fd3704db node DatanodeRegistration(127.0.0.1:58631, datanodeUuid=4dea7652-e7d5-48ca-8292-8bb0be67b6f4, infoPort=32967, infoSecurePort=0, ipcPort=53953, storageInfo=lv=-56;cid=testClusterID;nsid=564158542;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 910284 INFO  (IPC Server handler 8 on 49935) [    ] BlockStateChange BLOCK* processReport: from storage DS-66504a1e-29bf-4e0a-8452-7414a432e537 node DatanodeRegistration(127.0.0.1:58631, datanodeUuid=4dea7652-e7d5-48ca-8292-8bb0be67b6f4, infoPort=32967, infoSecurePort=0, ipcPort=53953, storageInfo=lv=-56;cid=testClusterID;nsid=564158542;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 910370 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 910370 INFO  (Thread-3535) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 910370 INFO  (Thread-3535) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 910372 ERROR (Thread-3535) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 910470 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.ZkTestServer start zk server on port:41571
   [junit4]   2> 910485 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 910486 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 910488 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 910489 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 910491 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 910492 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 910493 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 910494 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 910495 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 910496 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 910497 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 910498 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 910591 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 910592 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5454954d{/,null,AVAILABLE}
   [junit4]   2> 910592 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@273bc2d4{HTTP/1.1,[http/1.1]}{127.0.0.1:59198}
   [junit4]   2> 910592 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.Server Started @912548ms
   [junit4]   2> 910592 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=59198, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/control-001/cores}
   [junit4]   2> 910593 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 910593 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 910594 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 910594 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 910594 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-20T09:39:20.038Z
   [junit4]   2> 910596 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 910597 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/control-001/solr.xml
   [junit4]   2> 910604 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 910607 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41571/solr
   [junit4]   2> 910675 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:59198_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 910675 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:59198_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59198_
   [junit4]   2> 910676 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:59198_    ] o.a.s.c.Overseer Overseer (id=98339622736035844-127.0.0.1:59198_-n_0000000000) starting
   [junit4]   2> 910681 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:59198_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59198_
   [junit4]   2> 910683 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 910843 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:59198_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 910847 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:59198_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 910847 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:59198_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 910848 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:59198_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/control-001/cores
   [junit4]   2> 910862 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 910863 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41571/solr ready
   [junit4]   2> 910865 INFO  (qtp1620821296-8377) [n:127.0.0.1:59198_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:59198_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 910867 INFO  (OverseerThreadFactory-2760-thread-1-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 910972 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 910972 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 911077 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 911989 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 912000 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 912075 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 912094 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 912095 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 912095 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49935/solr_hdfs_home
   [junit4]   2> 912095 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 912095 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 912095 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://localhost:49935/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 912096 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 912103 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 912103 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 912103 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 912107 WARN  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 917113 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 917114 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 917129 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 917135 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 917135 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 917138 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 917139 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@28780275
   [junit4]   2> 917159 INFO  (IPC Server handler 4 on 49935) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:44594 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b0216fe9-73f7-4d12-8c5e-d4a4fd3704db:NORMAL:127.0.0.1:58631|RBW], ReplicaUC[[DISK]DS-44f2195a-d4d5-4b95-806f-bb32c87db2ba:NORMAL:127.0.0.1:44594|FINALIZED]]} size 0
   [junit4]   2> 917160 INFO  (IPC Server handler 5 on 49935) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:58631 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b0216fe9-73f7-4d12-8c5e-d4a4fd3704db:NORMAL:127.0.0.1:58631|RBW], ReplicaUC[[DISK]DS-44f2195a-d4d5-4b95-806f-bb32c87db2ba:NORMAL:127.0.0.1:44594|FINALIZED]]} size 0
   [junit4]   2> 917166 WARN  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 917202 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 917202 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 917202 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 917211 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 917211 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 917213 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=33.66796875, floorSegmentMB=1.0263671875, forceMergeDeletesPctAllowed=27.618166472339453, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8267606389058317
   [junit4]   2> 917221 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@20e84152[control_collection_shard1_replica_n1] main]
   [junit4]   2> 917222 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 917222 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 917223 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 917224 INFO  (searcherExecutor-2763-thread-1-processing-n:127.0.0.1:59198_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@20e84152[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 917224 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573433970962464768
   [junit4]   2> 917230 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 917230 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 917231 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:59198/control_collection_shard1_replica_n1/
   [junit4]   2> 917231 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 917231 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:59198/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 917231 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 917232 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 917235 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:59198/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 917336 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 917386 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 917388 INFO  (qtp1620821296-8379) [n:127.0.0.1:59198_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=6416
   [junit4]   2> 917391 INFO  (qtp1620821296-8377) [n:127.0.0.1:59198_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 917491 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 918391 INFO  (qtp1620821296-8377) [n:127.0.0.1:59198_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:59198_&wt=javabin&version=2} status=0 QTime=7526
   [junit4]   2> 918398 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 918399 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41571/solr ready
   [junit4]   2> 918399 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 918401 INFO  (qtp1620821296-8374) [n:127.0.0.1:59198_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 918403 INFO  (OverseerThreadFactory-2760-thread-2-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 918404 INFO  (OverseerCollectionConfigSetProcessor-98339622736035844-127.0.0.1:59198_-n_0000000000) [n:127.0.0.1:59198_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 918404 WARN  (OverseerThreadFactory-2760-thread-2-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 918611 INFO  (qtp1620821296-8374) [n:127.0.0.1:59198_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 918614 INFO  (qtp1620821296-8374) [n:127.0.0.1:59198_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=212
   [junit4]   2> 918732 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-1-001 of type NRT
   [junit4]   2> 918733 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 918734 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7c580b68{/,null,AVAILABLE}
   [junit4]   2> 918734 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5ba1fd1d{HTTP/1.1,[http/1.1]}{127.0.0.1:56675}
   [junit4]   2> 918735 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.Server Started @920691ms
   [junit4]   2> 918735 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=56675, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-1-001/cores}
   [junit4]   2> 918735 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 918736 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 918736 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 918736 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 918741 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-20T09:39:28.180Z
   [junit4]   2> 918747 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 918747 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-1-001/solr.xml
   [junit4]   2> 918752 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 918759 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41571/solr
   [junit4]   2> 918766 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 918767 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:56675_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 918769 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:56675_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56675_
   [junit4]   2> 918770 INFO  (zkCallback-1002-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 918773 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 918774 INFO  (zkCallback-1007-thread-1-processing-n:127.0.0.1:56675_) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 918826 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:56675_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 918830 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:56675_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 918831 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:56675_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 918832 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:56675_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-1-001/cores
   [junit4]   2> 918852 INFO  (qtp1620821296-8375) [n:127.0.0.1:59198_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:56675_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 918854 INFO  (OverseerCollectionConfigSetProcessor-98339622736035844-127.0.0.1:59198_-n_0000000000) [n:127.0.0.1:59198_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 918855 INFO  (OverseerThreadFactory-2760-thread-3-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:56675_ for creating new replica
   [junit4]   2> 918857 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 918862 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 918969 INFO  (zkCallback-1007-thread-1-processing-n:127.0.0.1:56675_) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 919878 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 919888 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=test
   [junit4]   2> 920029 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 920042 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 920042 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 920042 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49935/solr_hdfs_home
   [junit4]   2> 920042 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 920042 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 920042 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-1-001/cores/collection1_shard2_replica_n1], dataDir=[hdfs://localhost:49935/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 920043 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 920049 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 920050 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 920052 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 920053 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 920065 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 920070 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 920070 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 920072 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 920073 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@32c1dc01
   [junit4]   2> 920087 INFO  (IPC Server handler 6 on 49935) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:58631 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-44f2195a-d4d5-4b95-806f-bb32c87db2ba:NORMAL:127.0.0.1:44594|RBW], ReplicaUC[[DISK]DS-66504a1e-29bf-4e0a-8452-7414a432e537:NORMAL:127.0.0.1:58631|FINALIZED]]} size 0
   [junit4]   2> 920088 INFO  (IPC Server handler 7 on 49935) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:44594 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-66504a1e-29bf-4e0a-8452-7414a432e537:NORMAL:127.0.0.1:58631|FINALIZED], ReplicaUC[[DISK]DS-c136754a-c95c-4594-8701-6a06002f4aeb:NORMAL:127.0.0.1:44594|FINALIZED]]} size 0
   [junit4]   2> 920092 WARN  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 920129 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 920129 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 920129 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 920140 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 920140 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 920141 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=33.66796875, floorSegmentMB=1.0263671875, forceMergeDeletesPctAllowed=27.618166472339453, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8267606389058317
   [junit4]   2> 920160 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3bc442c9[collection1_shard2_replica_n1] main]
   [junit4]   2> 920161 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 920162 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 920162 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 920163 INFO  (searcherExecutor-2774-thread-1-processing-n:127.0.0.1:56675_ x:collection1_shard2_replica_n1 s:shard2 c:collection1) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@3bc442c9[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 920164 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573433974045278208
   [junit4]   2> 920169 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 920169 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 920169 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:56675/collection1_shard2_replica_n1/
   [junit4]   2> 920169 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 920169 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:56675/collection1_shard2_replica_n1/ has no replicas
   [junit4]   2> 920169 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 920174 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:56675/collection1_shard2_replica_n1/ shard2
   [junit4]   2> 920275 INFO  (zkCallback-1007-thread-1-processing-n:127.0.0.1:56675_) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 920325 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 920326 INFO  (qtp1408004217-8425) [n:127.0.0.1:56675_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1468
   [junit4]   2> 920328 INFO  (qtp1620821296-8375) [n:127.0.0.1:59198_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:56675_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1476
   [junit4]   2> 920414 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-2-001 of type NRT
   [junit4]   2> 920415 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 920416 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@255740d0{/,null,AVAILABLE}
   [junit4]   2> 920416 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@72a80a91{HTTP/1.1,[http/1.1]}{127.0.0.1:36944}
   [junit4]   2> 920416 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.Server Started @922372ms
   [junit4]   2> 920416 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=36944, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-2-001/cores}
   [junit4]   2> 920417 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 920418 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 920418 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 920418 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 920418 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-20T09:39:29.862Z
   [junit4]   2> 920420 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 920421 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-2-001/solr.xml
   [junit4]   2> 920424 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 920427 INFO  (zkCallback-1007-thread-1-processing-n:127.0.0.1:56675_) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 920428 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41571/solr
   [junit4]   2> 920439 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:36944_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 920440 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:36944_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 920444 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:36944_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36944_
   [junit4]   2> 920445 INFO  (zkCallback-1007-thread-1-processing-n:127.0.0.1:56675_) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 920446 INFO  (zkCallback-1002-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 920446 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 920447 INFO  (zkCallback-1013-thread-1-processing-n:127.0.0.1:36944_) [n:127.0.0.1:36944_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 920498 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:36944_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 920502 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:36944_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 920502 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:36944_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 920503 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:36944_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-2-001/cores
   [junit4]   2> 920531 INFO  (qtp821921468-8464) [n:127.0.0.1:36944_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:36944_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 920533 INFO  (OverseerCollectionConfigSetProcessor-98339622736035844-127.0.0.1:59198_-n_0000000000) [n:127.0.0.1:59198_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 920533 INFO  (OverseerThreadFactory-2760-thread-4-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:36944_ for creating new replica
   [junit4]   2> 920538 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n3&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 920538 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 920642 INFO  (zkCallback-1007-thread-1-processing-n:127.0.0.1:56675_) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 920642 INFO  (zkCallback-1013-thread-1-processing-n:127.0.0.1:36944_) [n:127.0.0.1:36944_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 921550 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 921560 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.IndexSchema [collection1_shard1_replica_n3] Schema name=test
   [junit4]   2> 921626 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 921637 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n3' using configuration from collection collection1, trusted=true
   [junit4]   2> 921637 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n3' (registry 'solr.core.collection1.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 921637 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49935/solr_hdfs_home
   [junit4]   2> 921637 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 921637 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 921637 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrCore [[collection1_shard1_replica_n3] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-2-001/cores/collection1_shard1_replica_n3], dataDir=[hdfs://localhost:49935/solr_hdfs_home/collection1/core_node4/data/]
   [junit4]   2> 921638 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/collection1/core_node4/data/snapshot_metadata
   [junit4]   2> 921644 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 921644 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 921649 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 921649 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/collection1/core_node4/data
   [junit4]   2> 921662 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/collection1/core_node4/data/index
   [junit4]   2> 921667 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 921667 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 921669 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 921669 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3c6bc59b
   [junit4]   2> 921678 INFO  (IPC Server handler 8 on 49935) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:58631 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c136754a-c95c-4594-8701-6a06002f4aeb:NORMAL:127.0.0.1:44594|RBW], ReplicaUC[[DISK]DS-b0216fe9-73f7-4d12-8c5e-d4a4fd3704db:NORMAL:127.0.0.1:58631|FINALIZED]]} size 0
   [junit4]   2> 921679 INFO  (IPC Server handler 7 on 49935) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:44594 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b0216fe9-73f7-4d12-8c5e-d4a4fd3704db:NORMAL:127.0.0.1:58631|FINALIZED], ReplicaUC[[DISK]DS-44f2195a-d4d5-4b95-806f-bb32c87db2ba:NORMAL:127.0.0.1:44594|FINALIZED]]} size 0
   [junit4]   2> 921684 WARN  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 921717 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 921717 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 921717 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 921725 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 921725 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 921728 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=33.66796875, floorSegmentMB=1.0263671875, forceMergeDeletesPctAllowed=27.618166472339453, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8267606389058317
   [junit4]   2> 921732 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.SolrIndexSearcher Opening [Searcher@3388312c[collection1_shard1_replica_n3] main]
   [junit4]   2> 921733 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 921733 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 921733 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 921735 INFO  (searcherExecutor-2785-thread-1-processing-n:127.0.0.1:36944_ x:collection1_shard1_replica_n3 s:shard1 c:collection1) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrCore [collection1_shard1_replica_n3] Registered new searcher Searcher@3388312c[collection1_shard1_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 921735 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573433975692591104
   [junit4]   2> 921740 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 921740 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 921740 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:36944/collection1_shard1_replica_n3/
   [junit4]   2> 921740 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 921740 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SyncStrategy http://127.0.0.1:36944/collection1_shard1_replica_n3/ has no replicas
   [junit4]   2> 921740 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 921742 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:36944/collection1_shard1_replica_n3/ shard1
   [junit4]   2> 921844 INFO  (zkCallback-1007-thread-1-processing-n:127.0.0.1:56675_) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 921844 INFO  (zkCallback-1013-thread-1-processing-n:127.0.0.1:36944_) [n:127.0.0.1:36944_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 921893 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 921894 INFO  (qtp821921468-8466) [n:127.0.0.1:36944_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n3&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1356
   [junit4]   2> 921896 INFO  (qtp821921468-8464) [n:127.0.0.1:36944_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:36944_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1365
   [junit4]   2> 921995 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-3-001 of type NRT
   [junit4]   2> 921995 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 921996 INFO  (zkCallback-1007-thread-1-processing-n:127.0.0.1:56675_) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 921996 INFO  (zkCallback-1013-thread-1-processing-n:127.0.0.1:36944_) [n:127.0.0.1:36944_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 921996 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@50169584{/,null,AVAILABLE}
   [junit4]   2> 921997 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@62ebf943{HTTP/1.1,[http/1.1]}{127.0.0.1:60829}
   [junit4]   2> 921997 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.e.j.s.Server Started @923953ms
   [junit4]   2> 921997 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=60829, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-3-001/cores}
   [junit4]   2> 921998 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 921998 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 921998 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 921998 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 921998 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-20T09:39:31.442Z
   [junit4]   2> 922000 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 922000 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-3-001/solr.xml
   [junit4]   2> 922004 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 922006 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41571/solr
   [junit4]   2> 922012 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:60829_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 922014 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:60829_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 922015 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:60829_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60829_
   [junit4]   2> 922016 INFO  (zkCallback-1013-thread-1-processing-n:127.0.0.1:36944_) [n:127.0.0.1:36944_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 922017 INFO  (zkCallback-1002-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 922017 INFO  (zkCallback-1007-thread-1-processing-n:127.0.0.1:56675_) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 922017 INFO  (zkCallback-995-thread-1-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 922018 INFO  (zkCallback-1019-thread-1-processing-n:127.0.0.1:60829_) [n:127.0.0.1:60829_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 922114 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:60829_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 922119 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:60829_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 922119 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:60829_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 922120 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[6B327E92B5F4887]) [n:127.0.0.1:60829_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-3-001/cores
   [junit4]   2> 922148 INFO  (qtp440654466-8501) [n:127.0.0.1:60829_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:60829_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 922149 INFO  (OverseerCollectionConfigSetProcessor-98339622736035844-127.0.0.1:59198_-n_0000000000) [n:127.0.0.1:59198_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 922150 INFO  (OverseerThreadFactory-2760-thread-5-processing-n:127.0.0.1:59198_) [n:127.0.0.1:59198_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:60829_ for creating new replica
   [junit4]   2> 922152 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n5&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 922152 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 922256 INFO  (zkCallback-1007-thread-1-processing-n:127.0.0.1:56675_) [n:127.0.0.1:56675_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 922256 INFO  (zkCallback-1013-thread-1-processing-n:127.0.0.1:36944_) [n:127.0.0.1:36944_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 922256 INFO  (zkCallback-1019-thread-1-processing-n:127.0.0.1:60829_) [n:127.0.0.1:60829_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 923164 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 923174 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.s.IndexSchema [collection1_shard2_replica_n5] Schema name=test
   [junit4]   2> 923238 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 923249 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n5' using configuration from collection collection1, trusted=true
   [junit4]   2> 923249 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n5' (registry 'solr.core.collection1.shard2.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff5497a
   [junit4]   2> 923250 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49935/solr_hdfs_home
   [junit4]   2> 923250 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 923250 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 923250 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.SolrCore [[collection1_shard2_replica_n5] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/shard-3-001/cores/collection1_shard2_replica_n5], dataDir=[hdfs://localhost:49935/solr_hdfs_home/collection1/core_node6/data/]
   [junit4]   2> 923251 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/collection1/core_node6/data/snapshot_metadata
   [junit4]   2> 923257 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 923257 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 923259 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 923260 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/collection1/core_node6/data
   [junit4]   2> 923272 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49935/solr_hdfs_home/collection1/core_node6/data/index
   [junit4]   2> 923277 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collection1_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 923277 INFO  (qtp440654466-8502) [n:127.0.0.1:60829_ c:collection1 s:shard2  x:collectio

[...truncated too long message...]

e/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:49935) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-2141385656-10.41.0.5-1500543558278 (Datanode Uuid 531c9885-8f86-4c31-942e-ce24f0d0d7c2) service to localhost/127.0.0.1:49935
   [junit4]   2> 1189835 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 1189937 ERROR (SUITE-SharedFSAutoReplicaFailoverTest-seed#[6B327E92B5F4887]-worker) [    ] o.a.h.m.l.MethodMetric Error invoking method getBlocksTotal
   [junit4]   2> java.lang.reflect.InvocationTargetException
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:401)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:194)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.hdfsFailoverAfterClass(SharedFSAutoReplicaFailoverTest.java:97)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_6B327E92B5F4887-001
   [junit4]   2> Jul 20, 2017 9:43:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 35 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false): {}, locale=es-CU, timezone=America/Mexico_City
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=164018760,total=521666560
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrCloudWithSecureImpersonation, TestDocumentBuilder, DistribDocExpirationUpdateProcessorTest, TestBulkSchemaAPI, TestCloudRecovery, TestJoin, TestRandomCollapseQParserPlugin, TestWordDelimiterFilterFactory, TestSimpleQParserPlugin, CdcrRequestHandlerTest, SolrCoreTest, RankQueryTest, TestSchemaNameResource, ZkNodePropsTest, DistributedVersionInfoTest, MetricUtilsTest, TestLeaderElectionWithEmptyReplica, TestManagedSchemaAPI, ClassificationUpdateProcessorTest, CurrencyFieldTypeTest, SolrIndexSplitterTest, ImplicitSnitchTest, TestCollationFieldDocValues, NodeMutatorTest, DebugComponentTest, TestFieldResource, ExternalFileFieldSortTest, SSLMigrationTest, TestPrepRecovery, TestSolrCLIRunExample, SyncSliceTest, TestAuthenticationFramework, TestCSVResponseWriter, TestManagedStopFilterFactory, OutputWriterTest, TestSolrQueryParser, TestReplicationHandlerBackup, TestNumericTerms32, TestRecoveryHdfs, TestWriterPerf, ChaosMonkeyNothingIsSafeTest, TestStressUserVersions, SolrRequestParserTest, UUIDFieldTest, SpellCheckCollatorTest, TestHighlightDedupGrouping, CoreMergeIndexesAdminHandlerTest, ResponseHeaderTest, TestLegacyFieldReuse, MigrateRouteKeyTest, TestCSVLoader, TestLazyCores, TestCloudJSONFacetJoinDomain, FieldMutatingUpdateProcessorTest, AddBlockUpdateTest, TestQuerySenderNoQuery, DocumentAnalysisRequestHandlerTest, ExitableDirectoryReaderTest, TestFastOutputStream, TestFieldCacheVsDocValues, TestComponentsName, SolrMetricReporterTest, WordBreakSolrSpellCheckerTest, BasicZkTest, TestFieldCacheReopen, TestPhraseSuggestions, TestSolrDeletionPolicy1, TestConfigsApi, TestCryptoKeys, SolrCloudReportersTest, SharedFSAutoReplicaFailoverTest]
   [junit4] Completed [230/728 (1!)] on J0 in 290.99s, 1 test, 1 failure <<< FAILURES!

[...truncated 44928 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 13 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/13/

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.analytics.NoFacetCloudTest

Error Message:
Could not find collection:collection1

Stack Trace:
java.lang.AssertionError: Could not find collection:collection1
        at __randomizedtesting.SeedInfo.seed([D28FD85806D2A9ED]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
        at org.apache.solr.analytics.AbstractAnalyticsStatsCloudTest.setupCluster(AbstractAnalyticsStatsCloudTest.java:76)
        at org.apache.solr.analytics.NoFacetCloudTest.populate(NoFacetCloudTest.java:62)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:35568/t/ek: ADDREPLICA failed to create replica

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:35568/t/ek: ADDREPLICA failed to create replica
        at __randomizedtesting.SeedInfo.seed([27B7CB5F9E2E9C1A:AFE3F48530D2F1E2]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:626)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:252)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.UnloadDistributedZkTest.testCoreUnloadAndLeaders(UnloadDistributedZkTest.java:186)
        at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
        at __randomizedtesting.SeedInfo.seed([27B7CB5F9E2E9C1A:AFE3F48530D2F1E2]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:246)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11594 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> 443318 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/init-core-data-001
   [junit4]   2> 443318 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 443318 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 443320 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 443320 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 443478 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 443483 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 443485 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 443493 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_42470_hdfs____.lrni4n/webapp
   [junit4]   2> 443812 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42470
   [junit4]   2> 443977 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 443979 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 443990 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_33515_datanode____.dkbjkw/webapp
   [junit4]   2> 444367 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33515
   [junit4]   2> 444438 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 444439 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 444450 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_57191_datanode____.7pxylu/webapp
   [junit4]   2> 444594 INFO  (IPC Server handler 3 on 49511) [    ] BlockStateChange BLOCK* processReport: from storage DS-f3697e64-f8e7-416a-813f-13ab96e28697 node DatanodeRegistration(127.0.0.1:45466, datanodeUuid=3ae35b4a-ed22-4f72-9203-2bc6fed49f22, infoPort=37084, infoSecurePort=0, ipcPort=34405, storageInfo=lv=-56;cid=testClusterID;nsid=1466201784;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 444594 INFO  (IPC Server handler 3 on 49511) [    ] BlockStateChange BLOCK* processReport: from storage DS-41b558bd-48e1-411a-9e8b-56e1d06aea49 node DatanodeRegistration(127.0.0.1:45466, datanodeUuid=3ae35b4a-ed22-4f72-9203-2bc6fed49f22, infoPort=37084, infoSecurePort=0, ipcPort=34405, storageInfo=lv=-56;cid=testClusterID;nsid=1466201784;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 444836 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[27B7CB5F9E2E9C1A]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:57191
   [junit4]   2> 445113 INFO  (IPC Server handler 4 on 49511) [    ] BlockStateChange BLOCK* processReport: from storage DS-85ee1151-b3f5-4160-acf3-233c3a81b720 node DatanodeRegistration(127.0.0.1:36080, datanodeUuid=254fd730-af6c-4645-b474-1a647f685d66, infoPort=60817, infoSecurePort=0, ipcPort=38854, storageInfo=lv=-56;cid=testClusterID;nsid=1466201784;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 445113 INFO  (IPC Server handler 4 on 49511) [    ] BlockStateChange BLOCK* processReport: from storage DS-9c18e694-a278-4e8f-a921-dfad264c5e9a node DatanodeRegistration(127.0.0.1:36080, datanodeUuid=254fd730-af6c-4645-b474-1a647f685d66, infoPort=60817, infoSecurePort=0, ipcPort=38854, storageInfo=lv=-56;cid=testClusterID;nsid=1466201784;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 445207 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 445207 INFO  (Thread-7159) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 445207 INFO  (Thread-7159) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 445215 ERROR (Thread-7159) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 445307 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.ZkTestServer start zk server on port:41948
   [junit4]   2> 445314 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15d646a777b0000, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 445330 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 445331 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 445334 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 445335 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 445336 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 445336 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 445337 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 445338 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 445339 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 445340 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 445341 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 445342 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 445423 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 445424 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1574a9c4{/,null,AVAILABLE}
   [junit4]   2> 445425 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@767d0e7d{HTTP/1.1,[http/1.1]}{127.0.0.1:33413}
   [junit4]   2> 445425 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.Server Started @447224ms
   [junit4]   2> 445431 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {replicaType=NRT, hostContext=/, hostPort=33413, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/control-001/cores}
   [junit4]   2> 445431 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 445431 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 445431 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 445431 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 445431 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-21T09:11:25.530Z
   [junit4]   2> 445435 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 445435 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/control-001/solr.xml
   [junit4]   2> 445442 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 445444 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41948/solr
   [junit4]   2> 445506 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:33413_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 445507 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:33413_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33413_
   [junit4]   2> 445507 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:33413_    ] o.a.s.c.Overseer Overseer (id=98345175305879556-127.0.0.1:33413_-n_0000000000) starting
   [junit4]   2> 445511 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:33413_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33413_
   [junit4]   2> 445513 INFO  (OverseerStateUpdate-98345175305879556-127.0.0.1:33413_-n_0000000000) [n:127.0.0.1:33413_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 445627 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:33413_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 445630 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:33413_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 445630 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:33413_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 445631 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:33413_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/control-001/cores
   [junit4]   2> 445671 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 445671 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41948/solr ready
   [junit4]   2> 445682 INFO  (qtp273920413-9386) [n:127.0.0.1:33413_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33413_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 445697 INFO  (OverseerThreadFactory-839-thread-1-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 445800 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 445800 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 445904 INFO  (zkCallback-318-thread-1-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 446816 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 446826 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 446892 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 446903 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 446903 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 446904 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49511/solr_hdfs_home
   [junit4]   2> 446904 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 446904 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 446904 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://localhost:49511/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 446905 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49511/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 446911 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 446911 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 446912 WARN  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 451917 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 451918 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49511/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 451934 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49511/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 451939 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 451939 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 451944 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 451944 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=57.4892578125, floorSegmentMB=0.580078125, forceMergeDeletesPctAllowed=20.644160481455927, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 451960 INFO  (IPC Server handler 9 on 49511) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36080 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-41b558bd-48e1-411a-9e8b-56e1d06aea49:NORMAL:127.0.0.1:45466|RBW], ReplicaUC[[DISK]DS-9c18e694-a278-4e8f-a921-dfad264c5e9a:NORMAL:127.0.0.1:36080|RBW]]} size 0
   [junit4]   2> 451961 INFO  (IPC Server handler 5 on 49511) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45466 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-41b558bd-48e1-411a-9e8b-56e1d06aea49:NORMAL:127.0.0.1:45466|RBW], ReplicaUC[[DISK]DS-9c18e694-a278-4e8f-a921-dfad264c5e9a:NORMAL:127.0.0.1:36080|RBW]]} size 0
   [junit4]   2> 451966 WARN  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 451999 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 451999 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 451999 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 452007 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 452007 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 452008 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3ddf7f5d
   [junit4]   2> 452013 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2fb419b6[control_collection_shard1_replica_n1] main]
   [junit4]   2> 452019 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 452019 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 452019 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 452022 INFO  (searcherExecutor-842-thread-1-processing-n:127.0.0.1:33413_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@2fb419b6[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 452022 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573522812039069696
   [junit4]   2> 452027 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 452027 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 452027 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33413/control_collection_shard1_replica_n1/
   [junit4]   2> 452028 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 452028 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33413/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 452028 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 452029 INFO  (zkCallback-318-thread-1-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 452031 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33413/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 452133 INFO  (zkCallback-318-thread-1-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 452182 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 452183 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=6382
   [junit4]   2> 452185 INFO  (qtp273920413-9386) [n:127.0.0.1:33413_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 452284 INFO  (zkCallback-318-thread-1-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 453186 INFO  (qtp273920413-9386) [n:127.0.0.1:33413_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33413_&wt=javabin&version=2} status=0 QTime=7503
   [junit4]   2> 453199 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 453200 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41948/solr ready
   [junit4]   2> 453200 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 453200 INFO  (qtp273920413-9384) [n:127.0.0.1:33413_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 453202 INFO  (OverseerThreadFactory-839-thread-2-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 453203 INFO  (OverseerCollectionConfigSetProcessor-98345175305879556-127.0.0.1:33413_-n_0000000000) [n:127.0.0.1:33413_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 453203 WARN  (OverseerThreadFactory-839-thread-2-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 453409 INFO  (qtp273920413-9384) [n:127.0.0.1:33413_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 453409 INFO  (qtp273920413-9384) [n:127.0.0.1:33413_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=208
   [junit4]   2> 453561 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-1-001 of type NRT
   [junit4]   2> 453562 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 453563 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7677bff7{/,null,AVAILABLE}
   [junit4]   2> 453563 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1b403a5b{HTTP/1.1,[http/1.1]}{127.0.0.1:46174}
   [junit4]   2> 453563 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.Server Started @455363ms
   [junit4]   2> 453563 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=46174, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-1-001/cores}
   [junit4]   2> 453563 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 453563 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 453563 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 453564 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 453564 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-21T09:11:33.663Z
   [junit4]   2> 453567 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 453567 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-1-001/solr.xml
   [junit4]   2> 453571 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 453573 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41948/solr
   [junit4]   2> 453579 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:46174_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 453582 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:46174_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 453584 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:46174_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46174_
   [junit4]   2> 453585 INFO  (zkCallback-325-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 453588 INFO  (zkCallback-318-thread-1-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 453591 INFO  (zkCallback-330-thread-1-processing-n:127.0.0.1:46174_) [n:127.0.0.1:46174_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 453710 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:46174_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 453714 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:46174_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 453714 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:46174_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 453715 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:46174_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-1-001/cores
   [junit4]   2> 453741 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:46174_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 453745 INFO  (OverseerCollectionConfigSetProcessor-98345175305879556-127.0.0.1:33413_-n_0000000000) [n:127.0.0.1:33413_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 453745 INFO  (OverseerThreadFactory-839-thread-3-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:46174_ for creating new replica
   [junit4]   2> 453748 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 453749 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 454765 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 454776 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=test
   [junit4]   2> 454850 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 454861 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 454861 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 454862 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49511/solr_hdfs_home
   [junit4]   2> 454862 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 454862 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 454862 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-1-001/cores/collection1_shard2_replica_n1], dataDir=[hdfs://localhost:49511/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 454863 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49511/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 454870 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 454870 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 454874 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 454874 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49511/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 454886 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49511/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 454891 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 454891 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 454892 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 454893 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=57.4892578125, floorSegmentMB=0.580078125, forceMergeDeletesPctAllowed=20.644160481455927, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 454923 INFO  (IPC Server handler 8 on 49511) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45466 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85ee1151-b3f5-4160-acf3-233c3a81b720:NORMAL:127.0.0.1:36080|RBW], ReplicaUC[[DISK]DS-f3697e64-f8e7-416a-813f-13ab96e28697:NORMAL:127.0.0.1:45466|FINALIZED]]} size 0
   [junit4]   2> 454924 INFO  (IPC Server handler 0 on 49511) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36080 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85ee1151-b3f5-4160-acf3-233c3a81b720:NORMAL:127.0.0.1:36080|RBW], ReplicaUC[[DISK]DS-f3697e64-f8e7-416a-813f-13ab96e28697:NORMAL:127.0.0.1:45466|FINALIZED]]} size 0
   [junit4]   2> 454927 WARN  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 454963 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 454964 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 454964 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 454972 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 454972 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 454973 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7031f071
   [junit4]   2> 454976 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@52027e53[collection1_shard2_replica_n1] main]
   [junit4]   2> 454977 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 454978 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 454978 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 454980 INFO  (searcherExecutor-853-thread-1-processing-n:127.0.0.1:46174_ x:collection1_shard2_replica_n1 s:shard2 c:collection1) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@52027e53[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 454980 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573522815140757504
   [junit4]   2> 454985 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 454985 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 454985 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46174/collection1_shard2_replica_n1/
   [junit4]   2> 454985 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 454985 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:46174/collection1_shard2_replica_n1/ has no replicas
   [junit4]   2> 454985 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 454988 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46174/collection1_shard2_replica_n1/ shard2
   [junit4]   2> 455140 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 455141 INFO  (qtp1618307956-9437) [n:127.0.0.1:46174_ c:collection1 s:shard2  x:collection1_shard2_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1392
   [junit4]   2> 455144 INFO  (qtp273920413-9388) [n:127.0.0.1:33413_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:46174_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1402
   [junit4]   2> 455253 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-2-001 of type NRT
   [junit4]   2> 455255 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 455256 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@374f5b1f{/,null,AVAILABLE}
   [junit4]   2> 455256 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5bac7b64{HTTP/1.1,[http/1.1]}{127.0.0.1:48917}
   [junit4]   2> 455256 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.Server Started @457056ms
   [junit4]   2> 455256 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=48917, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-2-001/cores}
   [junit4]   2> 455257 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 455257 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 455257 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 455257 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 455257 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-21T09:11:35.356Z
   [junit4]   2> 455259 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 455259 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-2-001/solr.xml
   [junit4]   2> 455263 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 455265 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41948/solr
   [junit4]   2> 455273 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:48917_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 455274 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:48917_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 455276 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:48917_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48917_
   [junit4]   2> 455277 INFO  (zkCallback-330-thread-1-processing-n:127.0.0.1:46174_) [n:127.0.0.1:46174_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 455277 INFO  (zkCallback-318-thread-1-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 455278 INFO  (zkCallback-325-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 455286 INFO  (zkCallback-336-thread-1-processing-n:127.0.0.1:48917_) [n:127.0.0.1:48917_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 455348 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:48917_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 455352 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:48917_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 455352 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:48917_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 455353 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:48917_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-2-001/cores
   [junit4]   2> 455374 INFO  (qtp273920413-9386) [n:127.0.0.1:33413_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:48917_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 455376 INFO  (OverseerCollectionConfigSetProcessor-98345175305879556-127.0.0.1:33413_-n_0000000000) [n:127.0.0.1:33413_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 455376 INFO  (OverseerThreadFactory-839-thread-4-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:48917_ for creating new replica
   [junit4]   2> 455379 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n3&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 455379 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 456393 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 456404 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.IndexSchema [collection1_shard1_replica_n3] Schema name=test
   [junit4]   2> 456468 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 456481 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n3' using configuration from collection collection1, trusted=true
   [junit4]   2> 456482 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n3' (registry 'solr.core.collection1.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 456482 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49511/solr_hdfs_home
   [junit4]   2> 456482 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 456482 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 456482 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrCore [[collection1_shard1_replica_n3] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-2-001/cores/collection1_shard1_replica_n3], dataDir=[hdfs://localhost:49511/solr_hdfs_home/collection1/core_node4/data/]
   [junit4]   2> 456483 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49511/solr_hdfs_home/collection1/core_node4/data/snapshot_metadata
   [junit4]   2> 456489 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 456489 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 456491 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 456492 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49511/solr_hdfs_home/collection1/core_node4/data
   [junit4]   2> 456504 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49511/solr_hdfs_home/collection1/core_node4/data/index
   [junit4]   2> 456509 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 456509 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 456511 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 456511 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=57.4892578125, floorSegmentMB=0.580078125, forceMergeDeletesPctAllowed=20.644160481455927, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 456518 INFO  (IPC Server handler 0 on 49511) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36080 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-41b558bd-48e1-411a-9e8b-56e1d06aea49:NORMAL:127.0.0.1:45466|RBW], ReplicaUC[[DISK]DS-9c18e694-a278-4e8f-a921-dfad264c5e9a:NORMAL:127.0.0.1:36080|FINALIZED]]} size 0
   [junit4]   2> 456518 INFO  (IPC Server handler 1 on 49511) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45466 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-41b558bd-48e1-411a-9e8b-56e1d06aea49:NORMAL:127.0.0.1:45466|RBW], ReplicaUC[[DISK]DS-9c18e694-a278-4e8f-a921-dfad264c5e9a:NORMAL:127.0.0.1:36080|FINALIZED]]} size 0
   [junit4]   2> 456525 WARN  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 456559 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 456559 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 456559 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 456567 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 456567 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 456569 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@405b6ba9
   [junit4]   2> 456572 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.SolrIndexSearcher Opening [Searcher@4b1e7748[collection1_shard1_replica_n3] main]
   [junit4]   2> 456574 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 456574 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 456575 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 456584 INFO  (searcherExecutor-864-thread-1-processing-n:127.0.0.1:48917_ x:collection1_shard1_replica_n3 s:shard1 c:collection1) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SolrCore [collection1_shard1_replica_n3] Registered new searcher Searcher@4b1e7748[collection1_shard1_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 456584 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573522816822673408
   [junit4]   2> 456591 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 456591 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 456591 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:48917/collection1_shard1_replica_n3/
   [junit4]   2> 456591 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 456592 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.SyncStrategy http://127.0.0.1:48917/collection1_shard1_replica_n3/ has no replicas
   [junit4]   2> 456592 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 456594 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:48917/collection1_shard1_replica_n3/ shard1
   [junit4]   2> 456745 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 456749 INFO  (qtp836647316-9475) [n:127.0.0.1:48917_ c:collection1 s:shard1  x:collection1_shard1_replica_n3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n3&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1370
   [junit4]   2> 456752 INFO  (qtp273920413-9386) [n:127.0.0.1:33413_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:48917_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1378
   [junit4]   2> 456840 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-3-001 of type NRT
   [junit4]   2> 456840 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 456841 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@28ef09d1{/,null,AVAILABLE}
   [junit4]   2> 456842 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@54c1c9bc{HTTP/1.1,[http/1.1]}{127.0.0.1:57754}
   [junit4]   2> 456842 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.e.j.s.Server Started @458641ms
   [junit4]   2> 456842 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=57754, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-3-001/cores}
   [junit4]   2> 456843 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 456844 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 456844 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 456844 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 456844 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-21T09:11:36.943Z
   [junit4]   2> 456846 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 456846 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-3-001/solr.xml
   [junit4]   2> 456850 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 456853 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41948/solr
   [junit4]   2> 456860 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:57754_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 456861 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:57754_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 456864 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:57754_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57754_
   [junit4]   2> 456865 INFO  (zkCallback-325-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 456866 INFO  (zkCallback-330-thread-1-processing-n:127.0.0.1:46174_) [n:127.0.0.1:46174_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 456866 INFO  (zkCallback-336-thread-1-processing-n:127.0.0.1:48917_) [n:127.0.0.1:48917_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 456866 INFO  (zkCallback-318-thread-1-processing-n:127.0.0.1:33413_) [n:127.0.0.1:33413_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 456867 INFO  (zkCallback-342-thread-1-processing-n:127.0.0.1:57754_) [n:127.0.0.1:57754_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 456985 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:57754_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 456988 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:57754_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 456989 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:57754_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@69da19dc
   [junit4]   2> 456990 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[27B7CB5F9E2E9C1A]) [n:127.0.0.1:57754_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_27B7CB5F9E2E9C1A-001/shard-3-001/cores
   [junit4]   2> 457015 INFO  (qtp273920413-9387) [n:127.0.0.1:33413_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:57754_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 457018 INFO  (OverseerCollectionConfigSetProcessor-98345175305879556-127.0.0.1:33413_-n_0000000000) [n:127.0.0.1:33413_    ] o.a.s.c.

[...truncated too long message...]

menode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> at org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.teardownClass(HdfsUnloadDistributedZkTest.java:46)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> ... 54 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_27B7CB5F9E2E9C1A-001
   [junit4]   2> Jul 21, 2017 10:04:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1591, maxMBSortInHeap=5.522946822626658, sim=RandomSimilarity(queryNorm=true): {}, locale=es-HN, timezone=America/Shiprock
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=2,free=302158440,total=523763712
   [junit4]   2> NOTE: All tests run in this JVM: [TestDFRSimilarityFactory, LeaderElectionTest, TestCloudSchemaless, SpellCheckCollatorWithCollapseTest, TestFilteredDocIdSet, OverseerRolesTest, TestSort, TestChildDocTransformer, PeerSyncReplicationTest, NumericFieldsTest, TestCloudInspectUtil, TestCopyFieldCollectionResource, TestRecovery, TestStressInPlaceUpdates, TestDistributedStatsComponentCardinality, TestCustomDocTransformer, XsltUpdateRequestHandlerTest, DistanceUnitsTest, ReplaceNodeTest, PreAnalyzedFieldTest, HardAutoCommitTest, TestPseudoReturnFields, TestSchemaSimilarityResource, TestTolerantSearch, TestUninvertingReader, TestUniqueKeyFieldResource, DeleteReplicaTest, UninvertDocValuesMergePolicyTest, TestExactStatsCache, TestXmlQParser, SegmentsInfoRequestHandlerTest, TestInitQParser, TestPKIAuthenticationPlugin, TestNamedUpdateProcessors, DateFieldTest, DistribCursorPagingTest, TestHdfsCloudBackupRestore, MultiTermTest, SuggesterFSTTest, TestUtils, SearchHandlerTest, LeaderInitiatedRecoveryOnShardRestartTest, TestSearcherReuse, TestStandardQParsers, TestRequestStatusCollectionAPI, SpellPossibilityIteratorTest, SortByFunctionTest, DistributedFacetPivotSmallTest, SubstringBytesRefFilterTest, TestGroupingSearch, SolrGangliaReporterTest, TestJmxIntegration, ResponseLogComponentTest, ZkStateWriterTest, ShardSplitTest, DocValuesMissingTest, TestAnalyzedSuggestions, TestNRTOpen, TestPolicyCloud, ConnectionManagerTest, JvmMetricsTest, SolrTestCaseJ4Test, InfixSuggestersTest, TestHdfsBackupRestoreCore, TestZkChroot, RequiredFieldsTest, TestIndexSearcher, ClusterStateTest, SolrInfoBeanTest, TestCloudPseudoReturnFields, TestComplexPhraseQParserPlugin, UnloadDistributedZkTest, MinimalSchemaTest, RulesTest, SolrMetricManagerTest, AnalyticsQueryTest, TestReload, TestSurroundQueryParser, TestRebalanceLeaders, TestScoreJoinQPScore, UpdateLogTest, TestFieldCollectionResource, SimplePostToolTest, BadIndexSchemaTest, TestImplicitCoreProperties, TestCodecSupport, DistributedDebugComponentTest, TestConfigSetsAPIZkFailure, TestDefaultStatsCache, HdfsChaosMonkeyNothingIsSafeTest, SolrSlf4jReporterTest, AutoCommitTest, OverriddenZkACLAndCredentialsProvidersTest, MultiThreadedOCPTest, CircularListTest, TestSolr4Spatial2, TestManagedSynonymGraphFilterFactory, TestDistribDocBasedVersion, TestSolrCoreProperties, TestLegacyTerms, RecoveryZkTest, SimpleMLTQParserTest, DirectUpdateHandlerOptimizeTest, AtomicUpdateProcessorFactoryTest, PathHierarchyTokenizerFactoryTest, TestStreamBody, HighlighterMaxOffsetTest, CoreAdminRequestStatusTest, OverseerTest, TermVectorComponentDistributedTest, DateMathParserTest, DistributedFacetPivotLargeTest, TestCollectionAPI, TestCollectionAPIs, DocumentBuilderTest, CdcrUpdateLogTest, TestOverriddenPrefixQueryForCustomFieldType, BasicDistributedZkTest, TestCloudPivotFacet, TestClusterProperties, TestConfigSetsAPIExclusivity, TestLeaderElectionWithEmptyReplica, TestLocalFSCloudBackupRestore, TestLockTree, TestMiniSolrCloudCluster, TestRandomFlRTGCloud, TestTolerantUpdateProcessorCloud, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZkTest, HdfsUnloadDistributedZkTest]
   [junit4] Completed [552/728 (2!)] on J2 in 41.00s, 1 test, 1 error <<< FAILURES!

[...truncated 3222 lines...]
   [junit4] Suite: org.apache.solr.analytics.NoFacetCloudTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/contrib/solr-analytics/test/J2/temp/solr.analytics.NoFacetCloudTest_D28FD85806D2A9ED-001/init-core-data-001
   [junit4]   2> Jul 21, 2017 10:21:36 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=4308, maxDocsPerChunk=7, blockSize=520), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=4308, blockSize=520)), sim=RandomSimilarity(queryNorm=true): {}, locale=ru, timezone=Europe/Mariehamn
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=158642840,total=283639808
   [junit4]   2> NOTE: All tests run in this JVM: [FunctionTest, NoFacetCloudTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=NoFacetCloudTest -Dtests.seed=D28FD85806D2A9ED -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ru -Dtests.timezone=Europe/Mariehamn -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J2 | NoFacetCloudTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Could not find collection:collection1
   [junit4]    > at __randomizedtesting.SeedInfo.seed([D28FD85806D2A9ED]:0)
   [junit4]    > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
   [junit4]    > at org.apache.solr.analytics.AbstractAnalyticsStatsCloudTest.setupCluster(AbstractAnalyticsStatsCloudTest.java:76)
   [junit4]    > at org.apache.solr.analytics.NoFacetCloudTest.populate(NoFacetCloudTest.java:62)
   [junit4]    > at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [16/17 (1!)] on J2 in 39.63s, 0 tests, 1 failure <<< FAILURES!

[...truncated 40665 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 14 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/14/

4 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=21142, name=Thread-8088, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=21142, name=Thread-8088, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at __randomizedtesting.SeedInfo.seed([231FA0220E882BBF:AB4B9FF8A0744647]:0)
Caused by: org.apache.solr.common.SolrException: Could not load collection from ZK: collection2
        at __randomizedtesting.SeedInfo.seed([231FA0220E882BBF]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1112)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:646)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:633)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection2/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1124)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1110)
        ... 7 more


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
ObjectTracker found 6 object(s) that were not released!!! [MDCAwareThreadPoolExecutor, SolrIndexSearcher, SolrCore, MMapDirectory, MMapDirectory, MMapDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:872)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.search.SolrIndexSearcher  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:323)  at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2033)  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2185)  at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1071)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:960)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1019)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:349)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:709)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:478)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:928)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 6 object(s) that were not released!!! [MDCAwareThreadPoolExecutor, SolrIndexSearcher, SolrCore, MMapDirectory, MMapDirectory, MMapDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:872)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.search.SolrIndexSearcher
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:323)
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2033)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2185)
        at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1071)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:960)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1019)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:349)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:709)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:478)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:928)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([231FA0220E882BBF]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:297)
        at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores:     1) Thread[id=62923, name=searcherExecutor-2494-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores:
   1) Thread[id=62923, name=searcherExecutor-2494-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([231FA0220E882BBF]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=62923, name=searcherExecutor-2494-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=62923, name=searcherExecutor-2494-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([231FA0220E882BBF]:0)




Build Log:
[...truncated 12612 lines...]
   [junit4] Suite: org.apache.solr.core.TestLazyCores
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/init-core-data-001
   [junit4]   2> 3920868 WARN  (SUITE-TestLazyCores-seed#[231FA0220E882BBF]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 3920868 INFO  (SUITE-TestLazyCores-seed#[231FA0220E882BBF]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 3920882 INFO  (SUITE-TestLazyCores-seed#[231FA0220E882BBF]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 3920882 INFO  (SUITE-TestLazyCores-seed#[231FA0220E882BBF]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3920884 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLazyLoad
   [junit4]   2> 3920967 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/solr.xml
   [junit4]   2> 3920984 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3921100 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3921217 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3921217 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3921243 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3921328 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3921358 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3921382 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3921492 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.s.IndexSchema [collection5] Schema name=tiny
   [junit4]   2> 3921494 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3921494 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.c.CoreContainer Creating SolrCore 'collection5' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/./collection5, trusted=true
   [junit4]   2> 3921495 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection5' (registry 'solr.core.collection5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3921525 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=tiny
   [junit4]   2> 3921526 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3921526 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/./collection1, trusted=true
   [junit4]   2> 3921527 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3921528 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.s.IndexSchema [collection2] Schema name=tiny
   [junit4]   2> 3921544 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3921544 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/./collection1/data/]
   [junit4]   2> 3921558 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3921558 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.c.SolrCore [[collection5] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/collection5], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/./collection5/data/]
   [junit4]   2> 3921567 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=97.6806640625, floorSegmentMB=1.3134765625, forceMergeDeletesPctAllowed=6.311377822195161, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3921574 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=97.6806640625, floorSegmentMB=1.3134765625, forceMergeDeletesPctAllowed=6.311377822195161, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3921589 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3921589 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.c.CoreContainer Creating SolrCore 'collection2' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/./collection2, trusted=true
   [junit4]   2> 3921589 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection2' (registry 'solr.core.collection2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3921589 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3921589 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.c.SolrCore [[collection2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/collection2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/./collection2/data/]
   [junit4]   2> 3921638 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=97.6806640625, floorSegmentMB=1.3134765625, forceMergeDeletesPctAllowed=6.311377822195161, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3921926 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3921926 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3921927 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3921928 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.s.SolrIndexSearcher Opening [Searcher@7eb16cfa[collection5] main]
   [junit4]   2> 3921928 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/collection5/conf
   [junit4]   2> 3921928 INFO  (coreLoadExecutor-2351-thread-3) [    x:collection5] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3921939 INFO  (searcherExecutor-2352-thread-1-processing-x:collection5) [    x:collection5] o.a.s.c.SolrCore [collection5] Registered new searcher Searcher@7eb16cfa[collection5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3921943 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3921943 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3921944 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3921944 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.s.SolrIndexSearcher Opening [Searcher@2cfaa6c7[collection2] main]
   [junit4]   2> 3921944 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/collection2/conf
   [junit4]   2> 3921945 INFO  (coreLoadExecutor-2351-thread-2) [    x:collection2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3921945 INFO  (searcherExecutor-2354-thread-1-processing-x:collection2) [    x:collection2] o.a.s.c.SolrCore [collection2] Registered new searcher Searcher@2cfaa6c7[collection2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3921991 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3921991 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3921992 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3921993 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6b85c42[collection1] main]
   [junit4]   2> 3921993 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/collection1/conf
   [junit4]   2> 3921993 INFO  (coreLoadExecutor-2351-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3922008 INFO  (searcherExecutor-2353-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6b85c42[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3922060 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3922091 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.s.IndexSchema [collection3] Schema name=tiny
   [junit4]   2> 3922105 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3922105 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.CoreContainer Creating SolrCore 'collection3' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/./collection3, trusted=true
   [junit4]   2> 3922106 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection3' (registry 'solr.core.collection3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3922106 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3922108 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore [[collection3] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/collection3], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/./collection3/data/]
   [junit4]   2> 3922138 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=2, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6833828385666243]
   [junit4]   2> 3922467 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3922467 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3922468 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@42493dff
   [junit4]   2> 3922468 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.s.SolrIndexSearcher Opening [Searcher@5cb6fd9f[collection3] main]
   [junit4]   2> 3922469 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/collection3/conf
   [junit4]   2> 3922469 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3922480 INFO  (searcherExecutor-2364-thread-1-processing-x:collection3) [    x:collection3] o.a.s.c.SolrCore [collection3] Registered new searcher Searcher@5cb6fd9f[collection3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3922535 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3922585 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.s.IndexSchema [collection4] Schema name=tiny
   [junit4]   2> 3922632 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3922632 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.CoreContainer Creating SolrCore 'collection4' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/./collection4, trusted=true
   [junit4]   2> 3922632 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection4' (registry 'solr.core.collection4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3922633 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3922633 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore [[collection4] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/collection4], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/./collection4/data/]
   [junit4]   2> 3922684 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=94.810546875, floorSegmentMB=2.0673828125, forceMergeDeletesPctAllowed=23.8915145912809, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7426421458816767
   [junit4]   2> 3922955 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3922955 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3922956 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3922956 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.s.SolrIndexSearcher Opening [Searcher@46c84d2d[collection4] main]
   [junit4]   2> 3922957 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-001/collection4/conf
   [junit4]   2> 3922957 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3922958 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2067830366
   [junit4]   2> 3922958 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 3923024 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 3923025 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 3923028 INFO  (coreCloseExecutor-2372-thread-1-processing-x:collection3) [    x:collection5] o.a.s.c.SolrCore [collection5]  CLOSING SolrCore org.apache.solr.core.SolrCore@426da8ae
   [junit4]   2> 3923044 INFO  (coreCloseExecutor-2372-thread-2-processing-x:collection3) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@790a9012
   [junit4]   2> 3923044 INFO  (coreCloseExecutor-2372-thread-3-processing-x:collection3) [    x:collection4] o.a.s.c.SolrCore [collection4]  CLOSING SolrCore org.apache.solr.core.SolrCore@18d7b56d
   [junit4]   2> 3923044 INFO  (coreCloseExecutor-2372-thread-4-processing-x:collection3) [    x:collection2] o.a.s.c.SolrCore [collection2]  CLOSING SolrCore org.apache.solr.core.SolrCore@243d7441
   [junit4]   2> 3923045 INFO  (coreCloseExecutor-2372-thread-5-processing-x:collection3) [    x:collection3] o.a.s.c.SolrCore [collection3]  CLOSING SolrCore org.apache.solr.core.SolrCore@4131d4fd
   [junit4]   2> 3923048 INFO  (searcherExecutor-2368-thread-1-processing-x:collection3) [    x:collection3] o.a.s.c.SolrCore [collection4] Registered new searcher Searcher@46c84d2d[collection4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3923060 INFO  (coreCloseExecutor-2372-thread-1-processing-x:collection3) [    x:collection5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection5, tag=1114482862
   [junit4]   2> 3923117 INFO  (coreCloseExecutor-2372-thread-4-processing-x:collection3) [    x:collection2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection2, tag=608007233
   [junit4]   2> 3923141 INFO  (coreCloseExecutor-2372-thread-2-processing-x:collection3) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=2030735378
   [junit4]   2> 3923198 INFO  (coreCloseExecutor-2372-thread-3-processing-x:collection3) [    x:collection4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection4, tag=416789869
   [junit4]   2> 3923222 INFO  (coreCloseExecutor-2372-thread-5-processing-x:collection3) [    x:collection3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection3, tag=1093784829
   [junit4]   2> 3923239 INFO  (TEST-TestLazyCores.testLazyLoad-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.SolrTestCaseJ4 ###Ending testLazyLoad
   [junit4]   2> 3923241 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.SolrTestCaseJ4 ###Starting testMidUseUnload
   [junit4]   2> 3923310 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/solr.xml
   [junit4]   2> 3923341 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3923390 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3923406 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3923406 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3923421 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3923477 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3923492 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3923532 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=tiny
   [junit4]   2> 3923535 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3923535 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/./collection1, trusted=true
   [junit4]   2> 3923536 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3923554 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.s.IndexSchema [collection2] Schema name=tiny
   [junit4]   2> 3923555 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3923555 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3923555 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/./collection1/data/]
   [junit4]   2> 3923557 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=97.6806640625, floorSegmentMB=1.3134765625, forceMergeDeletesPctAllowed=6.311377822195161, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3923559 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.s.IndexSchema [collection5] Schema name=tiny
   [junit4]   2> 3923568 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3923571 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.c.CoreContainer Creating SolrCore 'collection2' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/./collection2, trusted=true
   [junit4]   2> 3923572 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection2' (registry 'solr.core.collection2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3923572 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3923572 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.c.SolrCore [[collection2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/collection2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/./collection2/data/]
   [junit4]   2> 3923573 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3923573 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.c.CoreContainer Creating SolrCore 'collection5' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/./collection5, trusted=true
   [junit4]   2> 3923573 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection5' (registry 'solr.core.collection5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3923573 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3923574 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.c.SolrCore [[collection5] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/collection5], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/./collection5/data/]
   [junit4]   2> 3923578 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=97.6806640625, floorSegmentMB=1.3134765625, forceMergeDeletesPctAllowed=6.311377822195161, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3923592 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=97.6806640625, floorSegmentMB=1.3134765625, forceMergeDeletesPctAllowed=6.311377822195161, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3923742 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3923742 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3923743 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3923743 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@46906592[collection1] main]
   [junit4]   2> 3923744 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/collection1/conf
   [junit4]   2> 3923744 INFO  (coreLoadExecutor-2379-thread-1-processing-x:collection3) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3923759 INFO  (searcherExecutor-2380-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@46906592[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3923913 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3923913 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3923914 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3923946 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.s.SolrIndexSearcher Opening [Searcher@28a25757[collection2] main]
   [junit4]   2> 3923946 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/collection2/conf
   [junit4]   2> 3923946 INFO  (coreLoadExecutor-2379-thread-2-processing-x:collection3) [    x:collection2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3923950 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3923950 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3923951 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3923951 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.s.SolrIndexSearcher Opening [Searcher@60e56f1b[collection5] main]
   [junit4]   2> 3923951 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-002/collection5/conf
   [junit4]   2> 3923952 INFO  (coreLoadExecutor-2379-thread-3-processing-x:collection3) [    x:collection5] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3923954 INFO  (searcherExecutor-2382-thread-1-processing-x:collection5) [    x:collection5] o.a.s.c.SolrCore [collection5] Registered new searcher Searcher@60e56f1b[collection5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3923962 INFO  (searcherExecutor-2381-thread-1-processing-x:collection2) [    x:collection2] o.a.s.c.SolrCore [collection2] Registered new searcher Searcher@28a25757[collection2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3923986 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=null
   [junit4]   2> 3923997 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3924108 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3924209 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3924310 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3924429 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3924529 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3924641 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3924741 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3924841 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3924941 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3925047 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3925150 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3925251 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3925351 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3925451 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3925560 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3925661 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3925761 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3925873 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3925974 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3926076 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3926176 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3926276 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3926376 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3926491 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3926592 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3926694 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3926794 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3926894 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3926995 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3927096 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3927206 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3927308 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3927408 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3927519 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3927621 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3927721 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3927828 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3927938 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3928040 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3928140 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3928251 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3928353 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3928453 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 3928483 INFO  (Thread-51456) [    x:collection3] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2c688afb
   [junit4]   2> 3928499 INFO  (Thread-51456) [    x:collection3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=745048827
   [junit4]   2> 3928564 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1857155193
   [junit4]   2> 3928564 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 3928565 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 3928566 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 3928605 INFO  (coreCloseExecutor-2393-thread-1-processing-x:collection3) [    x:collection5] o.a.s.c.SolrCore [collection5]  CLOSING SolrCore org.apache.solr.core.SolrCore@7608b7f8
   [junit4]   2> 3928613 INFO  (coreCloseExecutor-2393-thread-2-processing-x:collection3) [    x:collection2] o.a.s.c.SolrCore [collection2]  CLOSING SolrCore org.apache.solr.core.SolrCore@207f5acb
   [junit4]   2> 3928649 INFO  (coreCloseExecutor-2393-thread-2-processing-x:collection3) [    x:collection2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection2, tag=545217227
   [junit4]   2> 3928651 INFO  (coreCloseExecutor-2393-thread-1-processing-x:collection3) [    x:collection5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection5, tag=1980282872
   [junit4]   2> 3928684 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.SolrTestCaseJ4 ###Ending testMidUseUnload
   [junit4]   2> 3928715 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.SolrTestCaseJ4 ###Starting testBadConfigsGenerateErrors
   [junit4]   2> 3928748 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3928821 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3928824 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3928824 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3928835 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.CorePropertiesLocator Found 6 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003
   [junit4]   2> 3928835 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.CorePropertiesLocator Cores are: [badConfig1, badSchema2, badSchema1, core2, badConfig2, core1]
   [junit4]   2> 3928835 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3928909 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:badConfig1] o.a.s.c.Config Ignoring exception matching \QSAXParseException\E
   [junit4]   2> 3928949 INFO  (coreLoadExecutor-2400-thread-2-processing-x:collection3) [    x:badSchema2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3928913 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:badSchema1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3928997 INFO  (coreLoadExecutor-2400-thread-2-processing-x:collection3) [    x:badSchema2] o.a.s.c.Config Ignoring exception matching \QSAXParseException\E
   [junit4]   2> 3928996 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:badSchema1] o.a.s.c.Config Ignoring exception matching \QSAXParseException\E
   [junit4]   2> 3928997 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3928999 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:badConfig2] o.a.s.c.Config Ignoring exception matching \QSAXParseException\E
   [junit4]   2> 3929028 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.s.IndexSchema [core2] Schema name=tiny
   [junit4]   2> 3929030 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3929030 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.c.CoreContainer Creating SolrCore 'core2' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/core2, trusted=true
   [junit4]   2> 3929030 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.core2' (registry 'solr.core.core2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3929030 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3929030 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.c.SolrCore [[core2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/core2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/core2/data/]
   [junit4]   2> 3929047 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3929060 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=97.6806640625, floorSegmentMB=1.3134765625, forceMergeDeletesPctAllowed=6.311377822195161, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3929065 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.s.IndexSchema [core1] Schema name=tiny
   [junit4]   2> 3929067 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3929067 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.c.CoreContainer Creating SolrCore 'core1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/core1, trusted=true
   [junit4]   2> 3929068 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.core1' (registry 'solr.core.core1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3929068 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3929068 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.c.SolrCore [[core1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/core1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/core1/data/]
   [junit4]   2> 3929138 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=97.6806640625, floorSegmentMB=1.3134765625, forceMergeDeletesPctAllowed=6.311377822195161, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3929438 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3929438 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3929439 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3929439 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.s.SolrIndexSearcher Opening [Searcher@1a062022[core2] main]
   [junit4]   2> 3929439 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/core2/conf
   [junit4]   2> 3929440 INFO  (coreLoadExecutor-2400-thread-1-processing-x:collection3) [    x:core2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3929441 INFO  (searcherExecutor-2401-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrCore [core2] Registered new searcher Searcher@1a062022[core2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3929539 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3929539 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3929540 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3929541 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.s.SolrIndexSearcher Opening [Searcher@13d5e93e[core1] main]
   [junit4]   2> 3929541 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/core1/conf
   [junit4]   2> 3929541 INFO  (coreLoadExecutor-2400-thread-3-processing-x:collection3) [    x:core1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3929542 INFO  (searcherExecutor-2402-thread-1-processing-x:core1) [    x:core1] o.a.s.c.SolrCore [core1] Registered new searcher Searcher@13d5e93e[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3929553 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 3929553 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@3058a744 commitCommandVersion:0
   [junit4]   2> 3929597 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.s.SolrIndexSearcher Opening [Searcher@2200049b[core1] main]
   [junit4]   2> 3929598 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 3929608 INFO  (searcherExecutor-2402-thread-1-processing-x:collection3) [    x:collection3] o.a.s.c.SolrCore [core1] Registered new searcher Searcher@2200049b[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c7)))}
   [junit4]   2> 3929624 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.S.Request [core1]  webapp=null path=null params={q={!prefix+f%3Dv_t}hel&wt=xml} hits=2 status=0 QTime=15
   [junit4]   2> 3929655 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.S.Request [core1]  webapp=null path=null params={q={!raw+f%3Dv_t}hello&wt=xml} hits=2 status=0 QTime=0
   [junit4]   2> 3929655 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.S.Request [core1]  webapp=null path=null params={q={!raw+f%3Dv_t}Hello&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 3929669 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.S.Request [core1]  webapp=null path=null params={q={!raw+f%3Dv_f}1.5&wt=xml} hits=0 status=0 QTime=14
   [junit4]   2> 3929789 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3929888 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.s.IndexSchema [badConfig1] Schema name=tiny
   [junit4]   2> 3929919 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3929919 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.CoreContainer Creating SolrCore 'badConfig1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/badConfig1, trusted=true
   [junit4]   2> 3929919 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.badConfig1' (registry 'solr.core.badConfig1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e4c77ee
   [junit4]   2> 3929919 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3929919 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.c.SolrCore [[badConfig1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/badConfig1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/badConfig1/data/]
   [junit4]   2> 3929921 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3930190 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3930190 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3930191 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7005e4d5
   [junit4]   2> 3930192 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.s.SolrIndexSearcher Opening [Searcher@617f0328[badConfig1] main]
   [junit4]   2> 3930192 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_231FA0220E882BBF-001/tempDir-003/badConfig1/conf
   [junit4]   2> 3930192 INFO  (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[231FA0220E882BBF]) [    x:collection3] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3930193 INFO  (searcherExecutor-2415-thread-1-processing-x:collection3) [    x:collection3] o.a.s.c.SolrCore [badConfig1] Registered new searcher Searcher@617f0328[badConfig1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3930225 INFO  (TEST-TestLazyCores.testBadConfigsGenerate

[...truncated too long message...]

ormat(Lucene50(blocksize=128))), 29parent_f1_s=PostingsFormat(name=Memory), 33parent_f1_s=PostingsFormat(name=Memory), 39parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 12parent_f1_s=PostingsFormat(name=Memory), 22parent_f1_s=PostingsFormat(name=Memory), 12grand1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 11grand6_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 2child1_s=PostingsFormat(name=Memory), 32parent_f1_s=FST50, 3parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 42parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 10grand3_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 6grand3_s=FST50, 31parent_f2_s=PostingsFormat(name=Memory), 41parent_f2_s=PostingsFormat(name=Memory), 11parent_f2_s=FST50, 21parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 7grand4_s=FST50, 3grand0_s=FST50, 8grand1_s=FST50, 9grand2_s=PostingsFormat(name=Memory), 9parent_f1_s=FST50, 14grand9_s=PostingsFormat(name=Memory), 14grand11_s=PostingsFormat(name=Memory), 13parent_f2_s=PostingsFormat(name=Memory), 23parent_f2_s=PostingsFormat(name=Memory), 8child5_s=PostingsFormat(name=Memory), 15grand1_s=FST50, 43parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 5child1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 14grand0_s=FST50, 33parent_f2_s=FST50, 13grand3_s=FST50, 22parent_f2_s=FST50, 4child0_s=PostingsFormat(name=Memory), 11grand5_s=FST50, 12grand2_s=FST50, 12parent_f2_s=PostingsFormat(name=Memory), 42parent_f2_s=PostingsFormat(name=Memory), 10grand4_s=FST50, 32parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 21parent_f1_s=FST50, 8grand2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 9grand1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 41parent_f1_s=PostingsFormat(name=Memory), 12grand8_s=PostingsFormat(name=Memory), 40parent_f2_s=FST50, 13grand7_s=FST50, 14parent_f1_s=FST50, 14grand10_s=PostingsFormat(name=Memory), 20parent_f2_s=PostingsFormat(name=Memory), 34parent_f1_s=PostingsFormat(name=Memory), 6child4_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 15grand9_s=FST50, 6grand0_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 7child3_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 15grand0_s=PostingsFormat(name=Memory), 8child6_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 5child0_s=FST50, 14grand1_s=PostingsFormat(name=Memory), 8parent_f1_s=PostingsFormat(name=Memory), 15grand13_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 27parent_f2_s=PostingsFormat(name=Memory), 2parent_f2_s=FST50, 4child1_s=PostingsFormat(name=Memory), 11grand4_s=PostingsFormat(name=Memory), 16parent_f2_s=PostingsFormat(name=Memory), 10grand5_s=PostingsFormat(name=Memory), 36parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 13grand10_s=PostingsFormat(name=Memory), 7grand6_s=PostingsFormat(name=Memory), 18parent_f1_s=PostingsFormat(name=Memory), 9grand8_s=FST50, 8grand3_s=PostingsFormat(name=Memory), 6parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 25parent_f1_s=FST50, 12grand9_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 0parent_f1_s=PostingsFormat(name=Memory), 3grand2_s=PostingsFormat(name=Memory), 9grand0_s=FST50, 4parent_f2_s=PostingsFormat(name=Memory), 13grand6_s=PostingsFormat(name=Memory), 6child5_s=FST50, 7child2_s=FST50, 38parent_f1_s=PostingsFormat(name=Memory), 5grand4_s=PostingsFormat(name=Memory), 6grand1_s=PostingsFormat(name=Memory), 15grand8_s=PostingsFormat(name=Memory), 8child7_s=FST50, 16parent_f1_s=FST50, 14grand2_s=PostingsFormat(name=Memory), 8parent_f2_s=PostingsFormat(name=Memory), 15grand14_s=FST50, 13grand5_s=PostingsFormat(name=Memory), 29parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 2parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 4child2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), rnd_b=PostingsFormat(name=Memory), 11grand3_s=PostingsFormat(name=Memory), 27parent_f1_s=PostingsFormat(name=Memory), 12grand0_s=PostingsFormat(name=Memory), 10grand6_s=PostingsFormat(name=Memory), 2child0_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 13grand11_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), 36parent_f1_s=FST50}, docValues:{_version_=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Lucene70), text_t=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=892, maxMBSortInHeap=7.277283683469657, sim=RandomSimilarity(queryNorm=true): {}, locale=sr-ME, timezone=US/Mountain
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=256017480,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [CollectionsAPIAsyncDistributedZkTest, ScriptEngineTest, TestCorePropertiesReload, TestComplexPhraseQParserPlugin, ClassificationUpdateProcessorIntegrationTest, SegmentsInfoRequestHandlerTest, TestOnReconnectListenerSupport, TestSolrIndexConfig, NoCacheHeaderTest, TestEmbeddedSolrServerSchemaAPI, TestRebalanceLeaders, DistributedTermsComponentTest, TestQueryTypes, TestSchemaSimilarityResource, TestSimpleTrackingShardHandler, SpellCheckComponentTest, TestRTGBase, MoreLikeThisHandlerTest, TestElisionMultitermQuery, V2ApiIntegrationTest, TestChildDocTransformer, ZkControllerTest, TestPerFieldSimilarity, SystemInfoHandlerTest, ResourceLoaderTest, TestRawResponseWriter, TestBulkSchemaConcurrent, TestUninvertingReader, TestReRankQParserPlugin, TestStandardQParsers, TestManagedSynonymGraphFilterFactory, TestImplicitCoreProperties, TestLegacyFieldCache, HdfsRestartWhileUpdatingTest, DistributedDebugComponentTest, SolrMetricManagerTest, BasicAuthStandaloneTest, UtilsToolTest, TestRestoreCore, TestUpdate, JSONWriterTest, DOMUtilTest, CoreAdminRequestStatusTest, IndexSchemaTest, TestLegacyField, TestNonDefinedSimilarityFactory, TestTolerantUpdateProcessorCloud, LukeRequestHandlerTest, LoggingHandlerTest, TestUniqueKeyFieldResource, TestLocalFSCloudBackupRestore, TestScoreJoinQPScore, ShardRoutingCustomTest, DeleteNodeTest, DocValuesNotIndexedTest, TestCryptoKeys, ConfigSetsAPITest, ShardRoutingTest, JvmMetricsTest, TestSolrConfigHandlerCloud, TestPullReplicaErrorHandling, SoftAutoCommitTest, SchemaVersionSpecificBehaviorTest, TestCustomDocTransformer, BlockJoinFacetRandomTest, TestLeaderElectionWithEmptyReplica, DistributedFacetPivotSmallTest, TestGraphMLResponseWriter, TestCursorMarkWithoutUniqueKey, AlternateDirectoryTest, QueryParsingTest, TestLegacyTerms, TestSolrCloudWithKerberosAlt, TestNamedUpdateProcessors, TestClusterProperties, HighlighterMaxOffsetTest, TestCopyFieldCollectionResource, DistributedIntervalFacetingTest, PreAnalyzedFieldManagedSchemaCloudTest, DistribJoinFromCollectionTest, DistributedMLTComponentTest, TermVectorComponentTest, DirectSolrSpellCheckerTest, TestOverriddenPrefixQueryForCustomFieldType, IndexSchemaRuntimeFieldTest, TestPKIAuthenticationPlugin, BadComponentTest, DirectSolrConnectionTest, CollectionsAPIDistributedZkTest, DistributedQueryComponentOptimizationTest, TestSubQueryTransformerDistrib, HttpSolrCallGetCoreTest, ZkCLITest, OverseerTaskQueueTest, ChaosMonkeySafeLeaderTest, TestRestManager, DirectoryFactoryTest, TriLevelCompositeIdRoutingTest, RemoteQueryErrorTest, SpatialFilterTest, TestJsonFacets, HdfsChaosMonkeyNothingIsSafeTest, CdcrUpdateLogTest, SparseHLLTest, TestFuzzyAnalyzedSuggestions, MoveReplicaTest, CSVRequestHandlerTest, TestFieldTypeResource, JsonLoaderTest, ZkStateWriterTest, TestExactStatsCache, TestFileDictionaryLookup, SubstringBytesRefFilterTest, OpenCloseCoreStressTest, TestAtomicUpdateErrorCases, TestFiltering, TestValueSourceCache, TestPartialUpdateDeduplication, HdfsNNFailoverTest, TestStressInPlaceUpdates, UnloadDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, BasicZkTest, FullSolrCloudDistribCmdsTest]
   [junit4] Completed [412/727 (2!)] on J1 in 1603.99s, 1 test, 1 error <<< FAILURES!

[...truncated 971 lines...]
   [junit4] JVM J1: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J1-20170722_143020_3243928966672887732899.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid14304.hprof ...
   [junit4] Heap dump file created [604151705 bytes in 12.396 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 7529 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:818: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:762: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid14304.hprof

Total time: 434 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 15 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/15/

7 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrBootstrapTest

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:478)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:928)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:908)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)  at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:484)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:332)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:419)  at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(CdcrRequestHandler.java:757)  at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(CdcrRequestHandler.java:712)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:349)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:709)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:908)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)  at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1019)  at org.apache.solr.core.SolrCore.reload(SolrCore.java:636)  at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1205)  at org.apache.solr.handler.IndexFetcher.lambda$reloadCore$0(IndexFetcher.java:900)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:908)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)  at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 5 object(s) that were not released!!! [MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:478)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:928)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:908)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
        at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:484)
        at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:332)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:419)
        at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(CdcrRequestHandler.java:757)
        at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(CdcrRequestHandler.java:712)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:349)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:709)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:908)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
        at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1019)
        at org.apache.solr.core.SolrCore.reload(SolrCore.java:636)
        at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1205)
        at org.apache.solr.handler.IndexFetcher.lambda$reloadCore$0(IndexFetcher.java:900)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:908)
        at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
        at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
        at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([8303D5389BDC5EF3]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:297)
        at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) Thread[id=78322, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    2) Thread[id=78379, name=zkCallback-2600-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=78344, name=StoppableCommitThread-SendThread(127.0.0.1:44507), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    4) Thread[id=78321, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    5) Thread[id=78345, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    6) Thread[id=78346, name=zkCallback-2600-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=78380, name=zkCallback-2600-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    8) Thread[id=78323, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
   1) Thread[id=78322, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   2) Thread[id=78379, name=zkCallback-2600-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=78344, name=StoppableCommitThread-SendThread(127.0.0.1:44507), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   4) Thread[id=78321, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   5) Thread[id=78345, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   6) Thread[id=78346, name=zkCallback-2600-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=78380, name=zkCallback-2600-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   8) Thread[id=78323, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
        at __randomizedtesting.SeedInfo.seed([8303D5389BDC5EF3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=78322, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Throwable.fillInStackTrace(Native Method)         at java.lang.Throwable.fillInStackTrace(Throwable.java:783)         at java.lang.Throwable.<init>(Throwable.java:265)         at java.lang.Exception.<init>(Exception.java:66)         at java.lang.RuntimeException.<init>(RuntimeException.java:62)         at java.lang.IllegalStateException.<init>(IllegalStateException.java:55)         at org.apache.http.util.Asserts.check(Asserts.java:34)         at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)         at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251)         at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175)         at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)         at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)         at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)         at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)         at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)         at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)         at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:523)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:252)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    2) Thread[id=78379, name=zkCallback-2600-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=78344, name=StoppableCommitThread-SendThread(127.0.0.1:44507), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    4) Thread[id=78321, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.io.PrintStream.write(PrintStream.java:478)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at java.io.PrintStream.newLine(PrintStream.java:546)         at java.io.PrintStream.println(PrintStream.java:824)         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at java.lang.Throwable.printStackTrace(Throwable.java:658)         at java.lang.Throwable.printStackTrace(Throwable.java:643)         at java.lang.Throwable.printStackTrace(Throwable.java:634)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)    5) Thread[id=78346, name=zkCallback-2600-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=78380, name=zkCallback-2600-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=78322, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Throwable.fillInStackTrace(Native Method)
        at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
        at java.lang.Throwable.<init>(Throwable.java:265)
        at java.lang.Exception.<init>(Exception.java:66)
        at java.lang.RuntimeException.<init>(RuntimeException.java:62)
        at java.lang.IllegalStateException.<init>(IllegalStateException.java:55)
        at org.apache.http.util.Asserts.check(Asserts.java:34)
        at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:523)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:252)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   2) Thread[id=78379, name=zkCallback-2600-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=78344, name=StoppableCommitThread-SendThread(127.0.0.1:44507), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   4) Thread[id=78321, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.io.PrintStream.write(PrintStream.java:478)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.newLine(PrintStream.java:546)
        at java.io.PrintStream.println(PrintStream.java:824)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
   5) Thread[id=78346, name=zkCallback-2600-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=78380, name=zkCallback-2600-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([8303D5389BDC5EF3]:0)


FAILED:  org.apache.solr.client.solrj.TestSolrJErrorHandling.testWithXml

Error Message:
expected:<998> but was:<996>

Stack Trace:
java.lang.AssertionError: expected:<998> but was:<996>
        at __randomizedtesting.SeedInfo.seed([BF64D64280B7C713:14866FB402B1DE7E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.client.solrj.TestSolrJErrorHandling.doThreads(TestSolrJErrorHandling.java:185)
        at org.apache.solr.client.solrj.TestSolrJErrorHandling.doIt(TestSolrJErrorHandling.java:200)
        at org.apache.solr.client.solrj.TestSolrJErrorHandling.testWithXml(TestSolrJErrorHandling.java:110)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap

Error Message:
Captured an uncaught exception in thread: Thread[id=139145, name=Thread-6584, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=139145, name=Thread-6584, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]
        at __randomizedtesting.SeedInfo.seed([8303D5389BDC5EF3:54D4FA4F2F83C6B4]:0)
Caused by: java.lang.AssertionError: 1
        at __randomizedtesting.SeedInfo.seed([8303D5389BDC5EF3]:0)
        at org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:192)
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1603)
        at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:832)
        at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1206)
        at org.apache.solr.handler.IndexFetcher.lambda$reloadCore$0(IndexFetcher.java:900)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([8303D5389BDC5EF3:B57EAE23520330B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
expected:<2> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<3>
        at __randomizedtesting.SeedInfo.seed([8303D5389BDC5EF3:B57EAE23520330B]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.assertSliceAndReplicaCount(SharedFSAutoReplicaFailoverTest.java:403)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:248)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11962 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_8303D5389BDC5EF3-001/init-core-data-001
   [junit4]   2> 1006689 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=3 numCloses=3
   [junit4]   2> 1006774 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1006775 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 1006775 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1006775 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1006995 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1007062 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1007068 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1007133 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_lucene2.us.west_apache_org_42381_hdfs____gs0sk8/webapp
   [junit4]   2> 1008187 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.m.log Started HttpServer2$[hidden email]:42381
   [junit4]   2> 1009317 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1009318 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1009360 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_39446_datanode____myx3zr/webapp
   [junit4]   2> 1010759 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39446
   [junit4]   2> 1011486 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1011487 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1011587 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46469_datanode____.vmmy5g/webapp
   [junit4]   2> 1012289 INFO  (IPC Server handler 3 on 46220) [    ] BlockStateChange BLOCK* processReport: from storage DS-dfbf2e62-1406-48ab-9637-698d7ef8d32d node DatanodeRegistration(127.0.0.1:46163, datanodeUuid=7fabdb3b-7af3-417e-8f5d-4f58940afbfe, infoPort=38924, infoSecurePort=0, ipcPort=34823, storageInfo=lv=-56;cid=testClusterID;nsid=741448241;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 1012289 INFO  (IPC Server handler 3 on 46220) [    ] BlockStateChange BLOCK* processReport: from storage DS-ef944c29-eaa4-4ca8-9b24-4659603eaad5 node DatanodeRegistration(127.0.0.1:46163, datanodeUuid=7fabdb3b-7af3-417e-8f5d-4f58940afbfe, infoPort=38924, infoSecurePort=0, ipcPort=34823, storageInfo=lv=-56;cid=testClusterID;nsid=741448241;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1012789 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[8303D5389BDC5EF3]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46469
   [junit4]   2> 1013583 INFO  (IPC Server handler 9 on 46220) [    ] BlockStateChange BLOCK* processReport: from storage DS-132fcc95-d0d9-4fdf-ae4f-f72d7f00d021 node DatanodeRegistration(127.0.0.1:42723, datanodeUuid=2461a33f-7fa7-4fd2-910c-ec0ced8dcae7, infoPort=38762, infoSecurePort=0, ipcPort=43034, storageInfo=lv=-56;cid=testClusterID;nsid=741448241;c=0), blocks: 0, hasStaleStorage: true, processing time: 20 msecs
   [junit4]   2> 1013583 INFO  (IPC Server handler 9 on 46220) [    ] BlockStateChange BLOCK* processReport: from storage DS-a1252db0-e3dd-480b-be30-7a556aeacd71 node DatanodeRegistration(127.0.0.1:42723, datanodeUuid=2461a33f-7fa7-4fd2-910c-ec0ced8dcae7, infoPort=38762, infoSecurePort=0, ipcPort=43034, storageInfo=lv=-56;cid=testClusterID;nsid=741448241;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1013898 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1013921 INFO  (Thread-3374) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1013921 INFO  (Thread-3374) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1013927 ERROR (Thread-3374) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1014026 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.ZkTestServer start zk server on port:34280
   [junit4]   2> 1014107 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1014125 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1014126 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1014127 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1014128 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1014128 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1014129 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1014130 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1014130 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1014131 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1014132 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1014154 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1014496 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1014497 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6261544a{/,null,AVAILABLE}
   [junit4]   2> 1014497 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@15167117{HTTP/1.1,[http/1.1]}{127.0.0.1:39167}
   [junit4]   2> 1014497 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.e.j.s.Server Started @1025219ms
   [junit4]   2> 1014497 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=39167, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_8303D5389BDC5EF3-001/control-001/cores}
   [junit4]   2> 1014497 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1014497 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 1014497 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1014497 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1014523 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-24T05:07:36.667Z
   [junit4]   2> 1014552 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1014552 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_8303D5389BDC5EF3-001/control-001/solr.xml
   [junit4]   2> 1014588 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@700ca18c, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1014595 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34280/solr
   [junit4]   2> 1014836 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [n:127.0.0.1:39167_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1014836 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [n:127.0.0.1:39167_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39167_
   [junit4]   2> 1014837 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [n:127.0.0.1:39167_    ] o.a.s.c.Overseer Overseer (id=98361203495141380-127.0.0.1:39167_-n_0000000000) starting
   [junit4]   2> 1014850 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [n:127.0.0.1:39167_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39167_
   [junit4]   2> 1014892 INFO  (zkCallback-313-thread-1-processing-n:127.0.0.1:39167_) [n:127.0.0.1:39167_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1015112 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [n:127.0.0.1:39167_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@700ca18c
   [junit4]   2> 1015132 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [n:127.0.0.1:39167_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@700ca18c
   [junit4]   2> 1015132 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [n:127.0.0.1:39167_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@700ca18c
   [junit4]   2> 1015133 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [n:127.0.0.1:39167_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_8303D5389BDC5EF3-001/control-001/cores
   [junit4]   2> 1015236 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1015254 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[8303D5389BDC5EF3]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34280/solr ready
   [junit4]   2> 1015266 INFO  (qtp688506182-5658) [n:127.0.0.1:39167_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:39167_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1015347 INFO  (OverseerThreadFactory-1061-thread-1-processing-n:127.0.0.1:39167_) [n:127.0.0.1:39167_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1015485 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1015485 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1015653 INFO  (zkCallback-313-thread-2-processing-n:127.0.0.1:39167_) [n:127.0.0.1:39167_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1019804 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1019848 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1020108 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1020338 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1020339 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@700ca18c
   [junit4]   2> 1020339 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://lucene2-us-west.apache.org:46220/solr_hdfs_home
   [junit4]   2> 1020339 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1020339 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1020339 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_8303D5389BDC5EF3-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://lucene2-us-west.apache.org:46220/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 1020341 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46220/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 1020383 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1020383 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 1020383 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 1020470 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1020471 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46220/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1020496 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://lucene2-us-west.apache.org:46220/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1020514 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1020514 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 1020569 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1020570 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=78.0771484375, floorSegmentMB=0.630859375, forceMergeDeletesPctAllowed=19.28436306747029, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1020731 INFO  (IPC Server handler 7 on 46220) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46163 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-132fcc95-d0d9-4fdf-ae4f-f72d7f00d021:NORMAL:127.0.0.1:42723|RBW], ReplicaUC[[DISK]DS-ef944c29-eaa4-4ca8-9b24-4659603eaad5:NORMAL:127.0.0.1:46163|RBW]]} size 0
   [junit4]   2> 1020733 INFO  (IPC Server handler 6 on 46220) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42723 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-132fcc95-d0d9-4fdf-ae4f-f72d7f00d021:NORMAL:127.0.0.1:42723|RBW], ReplicaUC[[DISK]DS-ef944c29-eaa4-4ca8-9b24-4659603eaad5:NORMAL:127.0.0.1:46163|RBW]]} size 0
   [junit4]   2> 1020781 WARN  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1020974 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1020986 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1020986 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 1021024 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1021024 INFO  (qtp688506182-5656) [n:127.0.0.1:39167_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled

[...truncated too long message...]

unit4]   2> 245469 INFO  (TEST-TestSolrJErrorHandling.testHttpURLConnection-seed#[BF64D64280B7C713]) [    ] o.a.s.c.s.TestSolrJErrorHandling CODE=400
   [junit4]   2> 245469 INFO  (TEST-TestSolrJErrorHandling.testHttpURLConnection-seed#[BF64D64280B7C713]) [    ] o.a.s.c.s.TestSolrJErrorHandling Attempting to get error stream.
   [junit4]   2> 245485 INFO  (TEST-TestSolrJErrorHandling.testHttpURLConnection-seed#[BF64D64280B7C713]) [    ] o.a.s.c.s.TestSolrJErrorHandling RESPONSE BODY:{"responseHeader":{"status":400,"QTime":0},"error":{"metadata":["error-class","org.apache.solr.common.SolrException","root-error-class","org.apache.solr.common.SolrException"],"msg":"ERROR: [doc=0] unknown field 'unknown_field_0'","code":400}}
   [junit4]   2>
   [junit4]   2> 245485 INFO  (TEST-TestSolrJErrorHandling.testHttpURLConnection-seed#[BF64D64280B7C713]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testHttpURLConnection
   [junit4]   2> 245503 INFO  (SUITE-TestSolrJErrorHandling-seed#[BF64D64280B7C713]-worker) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@74245252{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 245503 INFO  (SUITE-TestSolrJErrorHandling-seed#[BF64D64280B7C713]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=879646870
   [junit4]   2> 245504 INFO  (SUITE-TestSolrJErrorHandling-seed#[BF64D64280B7C713]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 245505 INFO  (SUITE-TestSolrJErrorHandling-seed#[BF64D64280B7C713]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 245506 INFO  (SUITE-TestSolrJErrorHandling-seed#[BF64D64280B7C713]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 245519 INFO  (SUITE-TestSolrJErrorHandling-seed#[BF64D64280B7C713]-worker) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 245528 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@81493ad
   [junit4]   2> 245528 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: now flush at close
   [junit4]   2> 245528 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]:   start flush: applyAllDeletes=true
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]:   index before flush _5(7.1.0):C489/488:delGen=1 _6(7.1.0):C2/2:delGen=1
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-491-thread-1]: startFullFlush
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-491-thread-1]: coreCloseExecutor-491-thread-1 finishFullFlush success=true
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: now apply all deletes for all segments buffered updates bytesUsed=0 reader pool bytesUsed=0
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][coreCloseExecutor-491-thread-1]: waitApply: no deletes to apply
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-491-thread-1]: updateMergeThreads ioThrottle=true targetMBPerSec=10240.0 MB/sec
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-491-thread-1]: now merge
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-491-thread-1]:   index: _5(7.1.0):C489/488:delGen=1 _6(7.1.0):C2/2:delGen=1
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-491-thread-1]:   no more merges pending; now return
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: waitForMerges
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: waitForMerges done
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: commit: start
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: commit: enter lock
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: commit: now prepare
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: prepareCommit: flush
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]:   index before flush _5(7.1.0):C489/488:delGen=1 _6(7.1.0):C2/2:delGen=1
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-491-thread-1]: startFullFlush
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: now apply all deletes for all segments buffered updates bytesUsed=0 reader pool bytesUsed=0
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [BD][coreCloseExecutor-491-thread-1]: waitApply: no deletes to apply
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-491-thread-1]: coreCloseExecutor-491-thread-1 finishFullFlush success=true
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: startCommit(): start
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]:   skip startCommit(): no changes pending
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: commit: pendingCommit == null; skip
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: commit: took 0.3 msec
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: commit: done
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: rollback
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: all running merges have aborted
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: rollback: done finish merges
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-491-thread-1]: abort
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-491-thread-1]: done abort success=true
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-491-thread-1]: rollback: infos=_5(7.1.0):C489/488:delGen=1 _6(7.1.0):C2/2:delGen=1
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-491-thread-1]: now checkpoint "_5(7.1.0):C489/488:delGen=1 _6(7.1.0):C2/2:delGen=1" [2 segments ; isCommit = false]
   [junit4]   2> 245529 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-491-thread-1]: 0 msec to checkpoint
   [junit4]   2> 245530 INFO  (coreCloseExecutor-491-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=135566253
   [junit4]   2> 245547 INFO  (SUITE-TestSolrJErrorHandling-seed#[BF64D64280B7C713]-worker) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@409e5b5b{/solr,null,UNAVAILABLE}
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestSolrJErrorHandling_BF64D64280B7C713-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=de-LU, timezone=PRC
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=97628792,total=312475648
   [junit4]   2> NOTE: All tests run in this JVM: [TestUpdateRequestCodec, TestCloudSolrClientConnections, QueryResponseTest, SchemaTest, StreamExpressionToExplanationTest, PowerEvaluatorTest, TestSolrProperties, TangentEvaluatorTest, TestConfigSetAdminRequest, SolrDocumentTest, SolrExampleEmbeddedTest, TestClusteringResponse, DivideEvaluatorTest, TestSolrJErrorHandling]
   [junit4] Completed [71/135 (1!)] on J1 in 69.37s, 4 tests, 1 failure <<< FAILURES!

[...truncated 42669 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 16 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/16/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
ObjectTracker found 3 object(s) that were not released!!! [RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1414)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:267)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:480)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1414)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:267)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:480)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1414)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:267)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:480)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 3 object(s) that were not released!!! [RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1414)
        at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:267)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:480)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1414)
        at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:267)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:480)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:972)
        at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1414)
        at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:267)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:480)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([B39E90D48F5B891]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:297)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.AutoCommitTest.testMaxDocs

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([B39E90D48F5B891:B2B83FD2641FBC1B]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:886)
        at org.apache.solr.update.AutoCommitTest.testMaxDocs(AutoCommitTest.java:225)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

        request was:q=id:14&qt=&start=0&rows=20&version=2.2
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:879)
        ... 40 more


FAILED:  org.apache.solr.search.join.TestScoreJoinQPNoScore.testJoin

Error Message:
mismatch: '1'!='4' @ response/docs/[0]/id

Stack Trace:
java.lang.RuntimeException: mismatch: '1'!='4' @ response/docs/[0]/id
        at __randomizedtesting.SeedInfo.seed([B39E90D48F5B891:360ADC8729A2BF07]:0)
        at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:974)
        at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:921)
        at org.apache.solr.search.join.TestScoreJoinQPNoScore.testJoin(TestScoreJoinQPNoScore.java:99)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
expected:<2> but was:<4>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<4>
        at __randomizedtesting.SeedInfo.seed([B39E90D48F5B891:836DD6D7E609D569]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.assertSliceAndReplicaCount(SharedFSAutoReplicaFailoverTest.java:403)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:248)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11324 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4]   2> 0    INFO  (SUITE-OpenCloseCoreStressTest-seed#[B39E90D48F5B891]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/init-core-data-001
   [junit4]   2> 290  INFO  (SUITE-OpenCloseCoreStressTest-seed#[B39E90D48F5B891]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 362  INFO  (SUITE-OpenCloseCoreStressTest-seed#[B39E90D48F5B891]-worker) [    ] o.e.j.u.log Logging initialized @2552ms
   [junit4]   2> 374  INFO  (SUITE-OpenCloseCoreStressTest-seed#[B39E90D48F5B891]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 759  INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test15Seconds
   [junit4]   2> 1074 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1104 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6dfbfc64{/solr,null,AVAILABLE}
   [junit4]   2> 1146 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4a498f5d{SSL,[ssl, http/1.1]}{127.0.0.1:42075}
   [junit4]   2> 1146 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.e.j.s.Server Started @3340ms
   [junit4]   2> 1147 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42075}
   [junit4]   2> 1242 ERROR (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1243 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 1243 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1243 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1265 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-25T10:30:38.009Z
   [junit4]   2> 1302 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1309 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/solr.xml
   [junit4]   2> 2223 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.c.CorePropertiesLocator Found 7 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001
   [junit4]   2> 2227 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [00000_core, 00004_core, 00003_core, 00002_core, 00006_core, 00001_core, 00005_core]
   [junit4]   2> 2228 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2336 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.c.OpenCloseCoreStressTest Starting indexing and querying
   [junit4]   2> 2338 INFO  (TEST-OpenCloseCoreStressTest.test15Seconds-seed#[B39E90D48F5B891]) [    ] o.a.s.c.OpenCloseCoreStressTest
   [junit4]   2>
   [junit4]   2>
   [junit4]   2>
   [junit4]   2>
   [junit4]   2> Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 2340 INFO  (Thread-11) [    ] o.a.s.c.OneIndexer Starting indexing thread: 45
   [junit4]   2> 2346 INFO  (Thread-12) [    ] o.a.s.c.OneIndexer Starting indexing thread: 46
   [junit4]   2> 2354 INFO  (Thread-13) [    ] o.a.s.c.OneIndexer Starting indexing thread: 47
   [junit4]   2> 2357 INFO  (Thread-14) [    ] o.a.s.c.OneIndexer Starting indexing thread: 48
   [junit4]   2> 2358 INFO  (Thread-16) [    ] o.a.s.c.OneIndexer Starting indexing thread: 50
   [junit4]   2> 2358 INFO  (Thread-15) [    ] o.a.s.c.OneIndexer Starting indexing thread: 49
   [junit4]   2> 2362 INFO  (Thread-17) [    ] o.a.s.c.OneIndexer Starting indexing thread: 51
   [junit4]   2> 2366 INFO  (Thread-18) [    ] o.a.s.c.OneIndexer Starting indexing thread: 52
   [junit4]   2> 2380 INFO  (Thread-19) [    ] o.a.s.c.OneIndexer Starting indexing thread: 53
   [junit4]   2> 2404 INFO  (Thread-20) [    ] o.a.s.c.OneQuery Starting query thread: 54
   [junit4]   2> 2415 INFO  (Thread-22) [    ] o.a.s.c.OneQuery Starting query thread: 56
   [junit4]   2> 2412 INFO  (Thread-21) [    ] o.a.s.c.OneQuery Starting query thread: 55
   [junit4]   2> 2422 INFO  (Thread-24) [    ] o.a.s.c.OneQuery Starting query thread: 58
   [junit4]   2> 2423 INFO  (Thread-25) [    ] o.a.s.c.OneQuery Starting query thread: 59
   [junit4]   2> 2424 INFO  (Thread-26) [    ] o.a.s.c.OneQuery Starting query thread: 60
   [junit4]   2> 2424 INFO  (Thread-27) [    ] o.a.s.c.OneQuery Starting query thread: 61
   [junit4]   2> 2424 INFO  (Thread-23) [    ] o.a.s.c.OneQuery Starting query thread: 57
   [junit4]   2> 2427 INFO  (Thread-28) [    ] o.a.s.c.OneQuery Starting query thread: 62
   [junit4]   2> 3454 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3457 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3460 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3463 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3466 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 3590 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.s.IndexSchema [00004_core] Schema name=tiny
   [junit4]   2> 3599 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.s.IndexSchema [00005_core] Schema name=tiny
   [junit4]   2> 3602 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.s.IndexSchema [00001_core] Schema name=tiny
   [junit4]   2> 3635 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.s.IndexSchema [00006_core] Schema name=tiny
   [junit4]   2> 3636 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.s.IndexSchema [00003_core] Schema name=tiny
   [junit4]   2> 3840 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3841 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.c.CoreContainer Creating SolrCore '00006_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00006_core, trusted=true
   [junit4]   2> 3842 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3843 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.c.CoreContainer Creating SolrCore '00005_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00005_core, trusted=true
   [junit4]   2> 3847 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3847 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.c.CoreContainer Creating SolrCore '00001_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00001_core, trusted=true
   [junit4]   2> 3851 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3851 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 3852 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.c.CoreContainer Creating SolrCore '00004_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00004_core, trusted=true
   [junit4]   2> 3851 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.c.CoreContainer Creating SolrCore '00003_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00003_core, trusted=true
   [junit4]   2> 3910 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3914 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3916 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3934 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3935 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3938 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.c.SolrCore [[00005_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00005_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00005_core/data/]
   [junit4]   2> 3943 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.c.SolrCore [[00004_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00004_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00004_core/data/]
   [junit4]   2> 3943 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.c.SolrCore [[00001_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00001_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00001_core/data/]
   [junit4]   2> 3943 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.c.SolrCore [[00003_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00003_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00003_core/data/]
   [junit4]   2> 3943 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.c.SolrCore [[00006_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00006_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00006_core/data/]
   [junit4]   2> 3986 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5109985032671833]
   [junit4]   2> 3990 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5109985032671833]
   [junit4]   2> 3992 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5109985032671833]
   [junit4]   2> 3986 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5109985032671833]
   [junit4]   2> 3987 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5109985032671833]
   [junit4]   2> 5059 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5060 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5069 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=16.6064453125, floorSegmentMB=0.3955078125, forceMergeDeletesPctAllowed=8.29832968520028, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13872188112827635
   [junit4]   2> 5061 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5102 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5066 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5103 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5104 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5104 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5104 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=16.6064453125, floorSegmentMB=0.3955078125, forceMergeDeletesPctAllowed=8.29832968520028, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13872188112827635
   [junit4]   2> 5106 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=16.6064453125, floorSegmentMB=0.3955078125, forceMergeDeletesPctAllowed=8.29832968520028, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13872188112827635
   [junit4]   2> 5107 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=16.6064453125, floorSegmentMB=0.3955078125, forceMergeDeletesPctAllowed=8.29832968520028, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13872188112827635
   [junit4]   2> 5110 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5110 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5117 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=16.6064453125, floorSegmentMB=0.3955078125, forceMergeDeletesPctAllowed=8.29832968520028, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13872188112827635
   [junit4]   2> 5130 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@66f8a05b[00005_core] main]
   [junit4]   2> 5134 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@c4bd5b0[00003_core] main]
   [junit4]   2> 5134 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@54e6ad40[00006_core] main]
   [junit4]   2> 5134 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@310d495e[00001_core] main]
   [junit4]   2> 5150 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00006_core/conf
   [junit4]   2> 5154 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00001_core/conf
   [junit4]   2> 5155 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00003_core/conf
   [junit4]   2> 5156 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00005_core/conf
   [junit4]   2> 5156 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@6c274faf[00004_core] main]
   [junit4]   2> 5157 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00004_core/conf
   [junit4]   2> 5218 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 5218 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 5226 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 5226 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 5218 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 5416 INFO  (searcherExecutor-8-thread-1-processing-x:00005_core) [    x:00005_core] o.a.s.c.SolrCore [00005_core] Registered new searcher Searcher@66f8a05b[00005_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 5416 INFO  (searcherExecutor-10-thread-1-processing-x:00006_core) [    x:00006_core] o.a.s.c.SolrCore [00006_core] Registered new searcher Searcher@54e6ad40[00006_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 5416 INFO  (searcherExecutor-9-thread-1-processing-x:00004_core) [    x:00004_core] o.a.s.c.SolrCore [00004_core] Registered new searcher Searcher@6c274faf[00004_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 5416 INFO  (searcherExecutor-7-thread-1-processing-x:00003_core) [    x:00003_core] o.a.s.c.SolrCore [00003_core] Registered new searcher Searcher@c4bd5b0[00003_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 5417 INFO  (searcherExecutor-11-thread-1-processing-x:00001_core) [    x:00001_core] o.a.s.c.SolrCore [00001_core] Registered new searcher Searcher@310d495e[00001_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 5512 INFO  (qtp286124372-14) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=2153
   [junit4]   2> 5514 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=2166
   [junit4]   2> 5538 INFO  (qtp286124372-15) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=2190
   [junit4]   2> 5514 INFO  (qtp286124372-70) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=2167
   [junit4]   2> 5514 INFO  (qtp286124372-71) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=2166
   [junit4]   2> 5514 INFO  (qtp286124372-17) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=2165
   [junit4]   2> 5514 INFO  (qtp286124372-67) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=2165
   [junit4]   2> 5514 INFO  (qtp286124372-64) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=2165
   [junit4]   2> 5514 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=2166
   [junit4]   2> 5659 INFO  (qtp286124372-63) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id8]} 0 2310
   [junit4]   2> 5666 INFO  (qtp286124372-74) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id9]} 0 2317
   [junit4]   2> 5666 INFO  (qtp286124372-68) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id1]} 0 2318
   [junit4]   2> 5708 INFO  (qtp286124372-75) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id5]} 0 2359
   [junit4]   2> 5673 INFO  (qtp286124372-21) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id2]} 0 2324
   [junit4]   2> 5673 INFO  (qtp286124372-19) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id4]} 0 2313
   [junit4]   2> 5682 INFO  (qtp286124372-20) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id6]} 0 2333
   [junit4]   2> 5682 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id7]} 0 2334
   [junit4]   2> 5713 INFO  (qtp286124372-70) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5714 INFO  (qtp286124372-73) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5715 INFO  (qtp286124372-69) [    x:00005_core] o.a.s.c.S.Request [00005_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5716 INFO  (qtp286124372-71) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5717 INFO  (qtp286124372-64) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5694 INFO  (qtp286124372-78) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id3]} 0 2345
   [junit4]   2> 5724 INFO  (qtp286124372-14) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id12]} 0 0
   [junit4]   2> 5731 INFO  (qtp286124372-76) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5731 INFO  (qtp286124372-74) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id16]} 0 0
   [junit4]   2> 5733 INFO  (qtp286124372-66) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id18]} 0 0
   [junit4]   2> 5748 INFO  (qtp286124372-72) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id10]} 0 81
   [junit4]   2> 5763 INFO  (qtp286124372-19) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id21]} 0 27
   [junit4]   2> 5778 INFO  (qtp286124372-75) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id15]} 0 45
   [junit4]   2> 5778 INFO  (qtp286124372-63) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id14]} 0 30
   [junit4]   2> 5781 INFO  (qtp286124372-64) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id24]} 0 0
   [junit4]   2> 5781 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 5783 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 5817 INFO  (qtp286124372-76) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5818 INFO  (qtp286124372-78) [    x:00005_core] o.a.s.c.S.Request [00005_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5820 INFO  (qtp286124372-66) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5821 INFO  (qtp286124372-78) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5821 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.s.IndexSchema [00002_core] Schema name=tiny
   [junit4]   2> 5826 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.s.IndexSchema [00000_core] Schema name=tiny
   [junit4]   2> 5834 INFO  (qtp286124372-71) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5862 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 5862 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.c.CoreContainer Creating SolrCore '00002_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00002_core, trusted=true
   [junit4]   2> 5862 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 5863 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.c.SolrCore [[00002_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00002_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00002_core/data/]
   [junit4]   2> 5865 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5109985032671833]
   [junit4]   2> 5868 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 5868 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.c.CoreContainer Creating SolrCore '00000_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00000_core, trusted=true
   [junit4]   2> 5869 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 5869 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.c.SolrCore [[00000_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00000_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00000_core/data/]
   [junit4]   2> 5870 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5109985032671833]
   [junit4]   2> 5946 INFO  (qtp286124372-63) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5948 INFO  (qtp286124372-75) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5950 INFO  (qtp286124372-63) [    x:00003_core] o.a.s.c.S.Request [00003_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 5985 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5985 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5991 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=16.6064453125, floorSegmentMB=0.3955078125, forceMergeDeletesPctAllowed=8.29832968520028, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13872188112827635
   [junit4]   2> 5992 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@30ea3d4b[00002_core] main]
   [junit4]   2> 5992 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00002_core/conf
   [junit4]   2> 5993 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 6002 INFO  (searcherExecutor-27-thread-1-processing-x:00002_core) [    x:00002_core] o.a.s.c.SolrCore [00002_core] Registered new searcher Searcher@30ea3d4b[00002_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 6003 INFO  (qtp286124372-63) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=52
   [junit4]   2> 6003 INFO  (qtp286124372-76) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=180
   [junit4]   2> 6004 INFO  (qtp286124372-72) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=49
   [junit4]   2> 6005 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=291
   [junit4]   2> 6022 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 6022 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 6025 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=16.6064453125, floorSegmentMB=0.3955078125, forceMergeDeletesPctAllowed=8.29832968520028, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13872188112827635
   [junit4]   2> 6025 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.s.SolrIndexSearcher Opening [Searcher@282c26e8[00000_core] main]
   [junit4]   2> 6026 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_B39E90D48F5B891-001/tempDir-001/00000_core/conf
   [junit4]   2> 6026 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 6034 INFO  (qtp286124372-19) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id23]} 0 259
   [junit4]   2> 6047 INFO  (searcherExecutor-28-thread-1-processing-x:00000_core) [    x:00000_core] o.a.s.c.SolrCore [00000_core] Registered new searcher Searcher@282c26e8[00000_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 6048 INFO  (qtp286124372-73) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id19]} 0 315
   [junit4]   2> 6048 INFO  (qtp286124372-17) [    x:00000_core] o.a.s.c.S.Request [00000_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=332
   [junit4]   2> 6048 INFO  (qtp286124372-18) [    x:00000_core] o.a.s.c.S.Request [00000_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=335
   [junit4]   2> 6049 INFO  (qtp286124372-66) [    x:00000_core] o.a.s.c.S.Request [00000_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 6050 INFO  (qtp286124372-68) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id13]} 0 318
   [junit4]   2> 6050 INFO  (qtp286124372-21) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id17]} 0 330
   [junit4]   2> 6051 INFO  (qtp286124372-78) [    x:00000_core] o.a.s.c.S.Request [00000_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 6047 INFO  (qtp286124372-69) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id20]} 0 285
   [junit4]   2> 6052 INFO  (qtp286124372-17) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id29]} 0 0
   [junit4]   2> 6052 INFO  (qtp286124372-74) [    x:00005_core] o.a.s.c.S.Request [00005_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 6053 INFO  (qtp286124372-75) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id27]} 0 13
   [junit4]   2> 6065 INFO  (qtp286124372-68) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id28]} 0 1
   [junit4]   2> 6067 INFO  (qtp286124372-64) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id26]} 0 285
   [junit4]   2> 6067 INFO  (qtp286124372-72) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id32]} 0 0
   [junit4]   2> 6072 INFO  (qtp286124372-69) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id30]} 0 18
   [junit4]   2> 6079 INFO  (qtp286124372-63) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id35]} 0 8
   [junit4]   2> 6079 INFO  (qtp286124372-72) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id33]} 0 10
   [junit4]   2> 6079 INFO  (qtp286124372-17) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id31]} 0 24
   [junit4]   2> 6080 INFO  (qtp286124372-19) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id36]} 0 0
   [junit4]   2> 6081 INFO  (qtp286124372-18) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id34]} 0 0
   [junit4]   2> 6081 INFO  (qtp286124372-66) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id38]} 0 0
   [junit4]   2> 6083 INFO  (qtp286124372-21) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id39]} 0 0
   [junit4]   2> 6083 INFO  (qtp286124372-19) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id40]} 0 0
   [junit4]   2> 6084 INFO  (qtp286124372-14) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id41]} 0 0
   [junit4]   2> 6085 INFO  (qtp286124372-19) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id42]} 0 0
   [junit4]   2> 6087 INFO  (qtp286124372-75) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id45]} 0 0
   [junit4]   2> 6088 INFO  (qtp286124372-19) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id44]} 0 0
   [junit4]   2> 6089 INFO  (qtp286124372-75) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id46]} 0 0
   [junit4]   2> 6091 INFO  (qtp286124372-67) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id47]} 0 0
   [junit4]   2> 6112 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 6113 INFO  (qtp286124372-19) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id48]} 0 0
   [junit4]   2> 6113 INFO  (qtp286124372-67) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 6114 INFO  (qtp286124372-67) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 6115 INFO  (qtp286124372-72) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 6116 INFO  (qtp286124372-15) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id11]} 0 386
   [junit4]   2> 6117 INFO  (qtp286124372-70) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id22]} 0 354
   [junit4]   2> 6118 INFO  (qtp286124372-73) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id37]} 0 0
   [junit4]   2> 6119 INFO  (qtp286124372-14) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id43]} 0 33
   [junit4]   2> 6120 INFO  (qtp286124372-21) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id49]} 0 28
   [junit4]   2> 6120 INFO  (qtp286124372-20) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id25]} 0 321
   [junit4]   2> 6120 INFO  (qtp286124372-64) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id50]} 0 6
   [junit4]   2> 6123 INFO  (qtp286124372-66) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id51]} 0 0
   [junit4]   2> 6124 INFO  (qtp286124372-66) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id53]} 0 0
   [junit4]   2> 6125 INFO  (qtp286124372-66) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id56]} 0 0
   [junit4]   2> 6127 INFO  (qtp286124372-71) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id60]} 0 0
   [junit4]   2> 6127 INFO  (qtp286124372-78) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id61]} 0 0
   [junit4]   2> 6129 INFO  (qtp286124372-71) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id62]} 0 0
   [junit4]   2> 6129 INFO  (qtp286124372-76) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id64]} 0 0
   [junit4]   2> 6131 INFO  (qtp286124372-19) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id52]} 0 9
   [junit4]   2> 6131 INFO  (qtp286124372-76) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id65]} 0 0
   [junit4]   2> 6132 INFO  (qtp286124372-19) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id66]} 0 0
   [junit4]   2> 6133 INFO  (qtp286124372-76) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id67]} 0 0
   [junit4]   2> 6133 INFO  (qtp286124372-71) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id63]} 0 4
   [junit4]   2> 6134 INFO  (qtp286124372-69) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id68]} 0 0
   [junit4]   2> 6135 INFO  (qtp286124372-76) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id70]} 0 0
   [junit4]   2> 6135 INFO  (qtp286124372-67) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id58]} 0 0
   [junit4]   2> 6135 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id69]} 0 0
   [junit4]   2> 6136 INFO  (qtp286124372-67) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id55]} 0 0
   [junit4]   2> 6136 INFO  (qtp286124372-69) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id57]} 0 0
   [junit4]   2> 6137 INFO  (qtp286124372-66) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id54]} 0 0
   [junit4]   2> 6137 INFO  (qtp286124372-67) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id59]} 0 0
   [junit4]   2> 6138 INFO  (qtp286124372-66) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id75]} 0 0
   [junit4]   2> 6140 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id74]} 0 0
   [junit4]   2> 6142 INFO  (qtp286124372-69) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id81]} 0 0
   [junit4]   2> 6143 INFO  (qtp286124372-15) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id73]} 0 0
   [junit4]   2> 6144 INFO  (qtp286124372-69) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id82]} 0 0
   [junit4]   2> 6144 INFO  (qtp286124372-15) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id78]} 0 0
   [junit4]   2> 6145 INFO  (qtp286124372-15) [    x:00003_core] o.a.s.u.p.LogUpdateProcessorFactory [00003_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id83]} 0 0
   [junit4]   2> 6146 INFO  (qtp286124372-15) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id85]} 0 0
   [junit4]   2> 6147 INFO  (qtp286124372-67) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id71]} 0 0
   [junit4]   2> 6147 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id76]} 0 0
   [junit4]   2> 6147 INFO  (qtp286124372-70) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id86]} 0 0
   [junit4]   2> 6148 INFO  (qtp286124372-73) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id79]} 0 0
   [junit4]   2> 6148 INFO  (qtp286124372-70) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id77]} 0 0
   [junit4]   2> 6149 INFO  (qtp286124372-15) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id87]} 0 0
   [junit4]   2> 6149 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id88]} 0 0
   [junit4]   2> 6151 INFO  (qtp286124372-15) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id91]} 0 0
   [junit4]   2> 6152 INFO  (qtp286124372-20) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 6151 INFO  (qtp286124372-21) [    x:00006_core] o.a.s.c.S.Request [00006_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 6151 INFO  (qtp286124372-14) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id90]} 0 0
   [junit4]   2> 6153 INFO  (qtp286124372-15) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id94]} 0 0
   [junit4]   2> 6154 INFO  (qtp286124372-73) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&w

[...truncated too long message...]

out/solr/build/solr-core/test/J1/temp/solr.search.join.TestScoreJoinQPNoScore_B39E90D48F5B891-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {small2_s_dv=BlockTreeOrds(blocksize=128), small2_ss_dv=PostingsFormat(name=LuceneVarGapDocFreqInterval), name_s=Lucene50(blocksize=128), dept_id_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), title_s_dv=PostingsFormat(name=LuceneVarGapDocFreqInterval), dept_ss=BlockTreeOrds(blocksize=128), name_s_dv=BlockTreeOrds(blocksize=128), title_s=PostingsFormat(name=LuceneVarGapFixedInterval), small_s_dv=BlockTreeOrds(blocksize=128), dept_ss_dv=Lucene50(blocksize=128), id=PostingsFormat(name=LuceneVarGapDocFreqInterval), text_t=BlockTreeOrds(blocksize=128), small3_ss_dv=BlockTreeOrds(blocksize=128), dept_id_s_dv=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{small2_s_dv=DocValuesFormat(name=Memory), id_dv=DocValuesFormat(name=Memory), small2_ss_dv=DocValuesFormat(name=Asserting), name_s=DocValuesFormat(name=Direct), title_s_dv=DocValuesFormat(name=Asserting), dept_ss=DocValuesFormat(name=Memory), name_s_dv=DocValuesFormat(name=Memory), title_s=DocValuesFormat(name=Lucene70), small_s_dv=DocValuesFormat(name=Memory), dept_ss_dv=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Asserting), small3_ss_dv=DocValuesFormat(name=Memory), dept_id_s_dv=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=837, maxMBSortInHeap=6.179037973942413, sim=RandomSimilarity(queryNorm=true): {}, locale=es-US, timezone=Atlantic/Bermuda
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=156683112,total=534773760
   [junit4]   2> NOTE: All tests run in this JVM: [OpenCloseCoreStressTest, DistributedDebugComponentTest, ZkStateWriterTest, ChaosMonkeySafeLeaderTest, BasicFunctionalityTest, SharedFSAutoReplicaFailoverUtilsTest, TestChildDocTransformer, TestPostingsSolrHighlighter, LargeFieldTest, CollectionReloadTest, TestConfigSetsAPI, TestCoreDiscovery, RulesTest, SOLR749Test, TestLegacyField, TestCustomDocTransformer, TestFieldCacheSort, TestManagedStopFilterFactory, HdfsTlogReplayBufferedWhileIndexingTest, ConfigSetsAPITest, SampleTest, TestSchemaVersionResource, SliceStateTest, TestSchemalessBufferedUpdates, StressHdfsTest, TestLocalFSCloudBackupRestore, TestNumericTerms64, TestCorePropertiesReload, TestTestInjection, TestValueSourceCache, TestSolrConfigHandlerCloud, ConvertedLegacyTest, TestEmbeddedSolrServerSchemaAPI, CloneFieldUpdateProcessorFactoryTest, TestRestManager, JSONWriterTest, BadCopyFieldTest, ConjunctionSolrSpellCheckerTest, TestSystemIdResolver, LeaderElectionContextKeyTest, TestJsonFacetRefinement, ZkSolrClientTest, OutOfBoxZkACLAndCredentialsProvidersTest, EnumFieldTest, TestSolrFieldCacheBean, DocValuesMultiTest, NoCacheHeaderTest, DistanceFunctionTest, TestRebalanceLeaders, TestTolerantUpdateProcessorCloud, AddSchemaFieldsUpdateProcessorFactoryTest, SpatialHeatmapFacetsTest, TestCryptoKeys, SuggesterTSTTest, CoreAdminHandlerTest, HdfsRestartWhileUpdatingTest, UUIDUpdateProcessorFallbackTest, TestShortCircuitedRequests, PrimUtilsTest, DOMUtilTest, TestHashPartitioner, DocValuesNotIndexedTest, CachingDirectoryFactoryTest, TestMaxScoreQueryParser, TestLegacyTerms, TestRTGBase, ParsingFieldUpdateProcessorsTest, TestPolicyCloud, TestSubQueryTransformerDistrib, SpatialFilterTest, TestSolrCloudWithHadoopAuthPlugin, TestDistributedGrouping, SegmentsInfoRequestHandlerTest, TestBlendedInfixSuggestions, TestUpdate, SystemInfoHandlerTest, ActionThrottleTest, CollectionsAPIDistributedZkTest, TestCloudPivotFacet, TestDFISimilarityFactory, TestManagedSchemaThreadSafety, SoftAutoCommitTest, TestHalfAndHalfDocValues, UpdateRequestProcessorFactoryTest, TestNoOpRegenerator, TestSchemaNameResource, TestFieldResource, TestFastLRUCache, TestSearchPerf, MoreLikeThisHandlerTest, TestStressInPlaceUpdates, OverseerModifyCollectionTest, TestDocTermOrds, TestObjectReleaseTracker, AnalyticsMergeStrategyTest, SolrGangliaReporterTest, ShardRoutingCustomTest, TestPullReplicaErrorHandling, SolrCloudExampleTest, InfixSuggestersTest, LukeRequestHandlerTest, LeaderInitiatedRecoveryOnCommitTest, CdcrVersionReplicationTest, TestExactSharedStatsCache, DistributedMLTComponentTest, TestQueryTypes, SubstringBytesRefFilterTest, TestSha256AuthenticationProvider, TestStreamBody, TestImplicitCoreProperties, CustomCollectionTest, GraphQueryTest, TestAtomicUpdateErrorCases, TestPivotHelperCode, TestPayloadScoreQParserPlugin, DirectSolrConnectionTest, ClusterStateUpdateTest, TestCursorMarkWithoutUniqueKey, OverseerTaskQueueTest, BasicDistributedZk2Test, SyncSliceTest, LeaderElectionIntegrationTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, TestRandomDVFaceting, ZkCLITest, TestDistributedSearch, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, TestSolrIndexConfig, ResponseLogComponentTest, BadComponentTest, TestStressRecovery, TestSuggestSpellingConverter, DateFieldTest, RAMDirectoryFactoryTest, TestSolrJ, TestLRUCache, ZkNodePropsTest, UUIDFieldTest, FileUtilsTest, CircularListTest, CursorPagingTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestDistributedMissingSort, TestSimpleTrackingShardHandler, TestTolerantSearch, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, ConnectionReuseTest, AsyncCallRequestStatusResponseTest, BaseCdcrDistributedZkTest, CdcrReplicationDistributedZkTest, CdcrReplicationHandlerTest, CollectionsAPIAsyncDistributedZkTest, ConcurrentDeleteAndCreateCollectionTest, DeleteLastCustomShardedReplicaTest, DeleteStatusTest, DistributedQueueTest, HttpPartitionTest, MoveReplicaHDFSTest, MultiThreadedOCPTest, RemoteQueryErrorTest, SaslZkACLProviderTest, SolrXmlInZkTest, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestCloudPseudoReturnFields, TestClusterProperties, TestCollectionAPI, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestDownShardTolerantSearch, TestHdfsCloudBackupRestore, TestLeaderElectionWithEmptyReplica, TestMiniSolrCloudClusterSSL, TestOnReconnectListenerSupport, TestRandomFlRTGCloud, TlogReplayBufferedWhileIndexingTest, HdfsRecoverLeaseTest, HdfsSyncSliceTest, BlobRepositoryCloudTest, DirectoryFactoryTest, ExitableDirectoryReaderTest, HdfsDirectoryFactoryTest, QueryResultKeyTest, TestBackupRepositoryFactory, TestConfigOverlay, TestConfigSetImmutable, TestConfigSetProperties, TestConfigSets, TestInfoStreamLogging, TestNRTOpen, TestReloadAndDeleteDocs, TestShardHandlerFactory, TestSimpleTextCodec, TestSolrConfigHandler, TestSolrXml, TestSolrCloudSnapshots, TestHdfsBackupRestoreCore, TestReplicationHandlerBackup, V2ApiIntegrationTest, V2StandaloneTest, CoreAdminRequestStatusTest, InfoHandlerTest, MetricsHandlerTest, PropertiesRequestHandlerTest, TestApiFramework, TestCollectionAPIs, TestCoreAdminApis, DistributedExpandComponentTest, DistributedFacetExistsSmallTest, DistributedFacetPivotLongTailTest, DistributedFacetPivotSmallAdvancedTest, DistributedFacetPivotWhiteBoxTest, ShufflingReplicaListTransformerTest, SuggestComponentTest, TestExpandComponent, TestHttpShardHandlerFactory, HighlighterMaxOffsetTest, TestUnifiedSolrHighlighter, TestSlowCompositeReaderWrapper, UninvertDocValuesMergePolicyTest, TestLegacyNumericUtils, TestNumericRangeQuery32, TestNumericRangeQuery64, JvmMetricsTest, MetricsConfigTest, SolrMetricManagerTest, SolrMetricReporterTest, SolrMetricsIntegrationTest, SolrGraphiteReporterTest, SolrSlf4jReporterTest, SolrCloudReportersTest, TestMacroExpander, TestExportWriter, TestGeoJSONResponseWriter, TestGraphMLResponseWriter, TestRawResponseWriter, TestRawTransformer, TestSubQueryTransformer, TestBulkSchemaAPI, TestCopyFieldCollectionResource, TestDynamicFieldCollectionResource, TestDynamicFieldResource, TestFieldTypeCollectionResource, TestFieldTypeResource, TestSchemaResource, TestUniqueKeyFieldResource, TestManagedSynonymGraphFilterFactory, ChangedSchemaMergeTest, CurrencyFieldTypeTest, DateRangeFieldTest, DocValuesMissingTest, ManagedSchemaRoundRobinCloudTest, SpatialRPTFieldTypeTest, TestCloudManagedSchema, TestSchemaManager, CursorMarkTest, MergeStrategyTest, RankQueryTest, SortSpecParsingTest, TestAddFieldRealTimeGet, TestCollapseQParserPlugin, TestComplexPhraseLeadingWildcard, TestComplexPhraseQParserPlugin, TestFieldSortValues, TestFilteredDocIdSet, TestHashQParserPlugin, TestMultiWordSynonyms, TestQueryWrapperFilter, TestRandomCollapseQParserPlugin, TestSimpleQParserPlugin, TestSmileRequest, TestStandardQParsers, TestTrieFacet, TestMinMaxOnMultiValuedField, TestOrdValues, BJQParserTest, BlockJoinFacetDistribTest, BlockJoinFacetRandomTest, BlockJoinFacetSimpleTest, TestCloudNestedDocsSort, TestScoreJoinQPNoScore]
   [junit4] Completed [669/725 (4!)] on J1 in 11.65s, 4 tests, 1 error <<< FAILURES!

[...truncated 179 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J1-20170725_103034_5234305422398238127463.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid7537.hprof ...
   [junit4] Heap dump file created [597571536 bytes in 2.693 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 7528 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:818: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:762: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid7537.hprof

Total time: 264 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 17 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/17/

5 tests failed.
FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at __randomizedtesting.SeedInfo.seed([DBC497A8C3B6A198:791CD972A744889E]:0)
        at org.apache.lucene.util.packed.Packed8ThreeBlocks.<init>(Packed8ThreeBlocks.java:41)
        at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:963)
        at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:939)
        at org.apache.lucene.util.packed.GrowableWriter.ensureCapacity(GrowableWriter.java:80)
        at org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:88)
        at org.apache.lucene.util.packed.AbstractPagedMutable.set(AbstractPagedMutable.java:98)
        at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:152)
        at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
        at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
        at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
        at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:296)
        at org.apache.lucene.util.fst.Builder.add(Builder.java:400)
        at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.writeFST(MemoryDocValuesConsumer.java:373)
        at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:416)
        at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:406)
        at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:527)
        at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:139)
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
        at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:181)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:125)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4392)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4032)
        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2235)
        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5050)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1720)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1462)
        at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2301)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([F5AA0A713AE9B538:7DFE35AB9415D8C0]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) Thread[id=10210, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    2) Thread[id=10492, name=zkCallback-1246-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=10491, name=zkCallback-1246-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=10490, name=zkCallback-1246-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=10212, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)         at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1124)         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1110)         at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:646)         at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)    6) Thread[id=10382, name=zkCallback-1246-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=10380, name=StoppableCommitThread-SendThread(127.0.0.1:34821), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    8) Thread[id=10211, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    9) Thread[id=10381, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
   1) Thread[id=10210, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   2) Thread[id=10492, name=zkCallback-1246-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=10491, name=zkCallback-1246-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=10490, name=zkCallback-1246-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=10212, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1124)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1110)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:646)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1227)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)
   6) Thread[id=10382, name=zkCallback-1246-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=10380, name=StoppableCommitThread-SendThread(127.0.0.1:34821), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   8) Thread[id=10211, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   9) Thread[id=10381, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
        at __randomizedtesting.SeedInfo.seed([F5AA0A713AE9B538]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=10210, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.security.AccessController.doPrivileged(Native Method)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)         at java.io.PrintStream.flush(PrintStream.java:338)         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)         at java.io.PrintStream.write(PrintStream.java:482)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at java.io.PrintStream.write(PrintStream.java:527)         at java.io.PrintStream.print(PrintStream.java:669)         at java.io.PrintStream.println(PrintStream.java:806)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)    2) Thread[id=10492, name=zkCallback-1246-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=10491, name=zkCallback-1246-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=10490, name=zkCallback-1246-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=10382, name=zkCallback-1246-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=10380, name=StoppableCommitThread-SendThread(127.0.0.1:34821), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    7) Thread[id=10211, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at org.apache.solr.common.util.ByteUtils.UTF16toUTF8(ByteUtils.java:90)         at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:840)         at org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:932)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=10210, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.security.AccessController.doPrivileged(Native Method)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.PrintStream.write(PrintStream.java:482)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.write(PrintStream.java:527)
        at java.io.PrintStream.print(PrintStream.java:669)
        at java.io.PrintStream.println(PrintStream.java:806)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
   2) Thread[id=10492, name=zkCallback-1246-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=10491, name=zkCallback-1246-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=10490, name=zkCallback-1246-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=10382, name=zkCallback-1246-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=10380, name=StoppableCommitThread-SendThread(127.0.0.1:34821), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   7) Thread[id=10211, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at org.apache.solr.common.util.ByteUtils.UTF16toUTF8(ByteUtils.java:90)
        at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:840)
        at org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:932)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
        at __randomizedtesting.SeedInfo.seed([F5AA0A713AE9B538]:0)


FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Didn't see all replicas for shard shard1 in c8n_1x2 come up within 90000 ms! ClusterState: {   "collMinRf_1x3":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{           "core_node4":{             "core":"collMinRf_1x3_shard1_replica_t2",             "base_url":"http://127.0.0.1:59492/y_dge/g",             "node_name":"127.0.0.1:59492_y_dge%2Fg",             "state":"active",             "type":"TLOG"},           "core_node5":{             "core":"collMinRf_1x3_shard1_replica_t1",             "base_url":"http://127.0.0.1:55265/y_dge/g",             "node_name":"127.0.0.1:55265_y_dge%2Fg",             "state":"active",             "type":"TLOG",             "leader":"true"},           "core_node6":{             "core":"collMinRf_1x3_shard1_replica_t3",             "base_url":"http://127.0.0.1:46521/y_dge/g",             "node_name":"127.0.0.1:46521_y_dge%2Fg",             "state":"active",             "type":"TLOG"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"0",     "tlogReplicas":"3"},   "collection1":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{       "shard1":{         "range":"80000000-ffffffff",         "state":"active",         "replicas":{"core_node4":{             "core":"collection1_shard1_replica_t3",             "base_url":"http://127.0.0.1:59492/y_dge/g",             "node_name":"127.0.0.1:59492_y_dge%2Fg",             "state":"active",             "type":"TLOG",             "leader":"true"}}},       "shard2":{         "range":"0-7fffffff",         "state":"active",         "replicas":{           "core_node2":{             "core":"collection1_shard2_replica_t1",             "base_url":"http://127.0.0.1:55265/y_dge/g",             "node_name":"127.0.0.1:55265_y_dge%2Fg",             "state":"active",             "type":"TLOG",             "leader":"true"},           "core_node6":{             "core":"collection1_shard2_replica_t5",             "base_url":"http://127.0.0.1:46521/y_dge/g",             "node_name":"127.0.0.1:46521_y_dge%2Fg",             "state":"active",             "type":"TLOG"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"},   "control_collection":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{"core_node2":{             "core":"control_collection_shard1_replica_n1",             "base_url":"http://127.0.0.1:36216/y_dge/g",             "node_name":"127.0.0.1:36216_y_dge%2Fg",             "state":"active",             "type":"NRT",             "leader":"true"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"1",     "tlogReplicas":"0"},   "c8n_1x2":{     "pullReplicas":"0",     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{           "core_node3":{             "core":"c8n_1x2_shard1_replica_t1",             "base_url":"http://127.0.0.1:55265/y_dge/g",             "node_name":"127.0.0.1:55265_y_dge%2Fg",             "state":"active",             "type":"TLOG",             "leader":"true"},           "core_node4":{             "state":"down",             "base_url":"http://127.0.0.1:59492/y_dge/g",             "core":"c8n_1x2_shard1_replica_t2",             "node_name":"127.0.0.1:59492_y_dge%2Fg",             "type":"TLOG"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "nrtReplicas":"0",     "tlogReplicas":"2"}}

Stack Trace:
java.lang.AssertionError: Didn't see all replicas for shard shard1 in c8n_1x2 come up within 90000 ms! ClusterState: {
  "collMinRf_1x3":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node4":{
            "core":"collMinRf_1x3_shard1_replica_t2",
            "base_url":"http://127.0.0.1:59492/y_dge/g",
            "node_name":"127.0.0.1:59492_y_dge%2Fg",
            "state":"active",
            "type":"TLOG"},
          "core_node5":{
            "core":"collMinRf_1x3_shard1_replica_t1",
            "base_url":"http://127.0.0.1:55265/y_dge/g",
            "node_name":"127.0.0.1:55265_y_dge%2Fg",
            "state":"active",
            "type":"TLOG",
            "leader":"true"},
          "core_node6":{
            "core":"collMinRf_1x3_shard1_replica_t3",
            "base_url":"http://127.0.0.1:46521/y_dge/g",
            "node_name":"127.0.0.1:46521_y_dge%2Fg",
            "state":"active",
            "type":"TLOG"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"0",
    "tlogReplicas":"3"},
  "collection1":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{
      "shard1":{
        "range":"80000000-ffffffff",
        "state":"active",
        "replicas":{"core_node4":{
            "core":"collection1_shard1_replica_t3",
            "base_url":"http://127.0.0.1:59492/y_dge/g",
            "node_name":"127.0.0.1:59492_y_dge%2Fg",
            "state":"active",
            "type":"TLOG",
            "leader":"true"}}},
      "shard2":{
        "range":"0-7fffffff",
        "state":"active",
        "replicas":{
          "core_node2":{
            "core":"collection1_shard2_replica_t1",
            "base_url":"http://127.0.0.1:55265/y_dge/g",
            "node_name":"127.0.0.1:55265_y_dge%2Fg",
            "state":"active",
            "type":"TLOG",
            "leader":"true"},
          "core_node6":{
            "core":"collection1_shard2_replica_t5",
            "base_url":"http://127.0.0.1:46521/y_dge/g",
            "node_name":"127.0.0.1:46521_y_dge%2Fg",
            "state":"active",
            "type":"TLOG"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "control_collection":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{"core_node2":{
            "core":"control_collection_shard1_replica_n1",
            "base_url":"http://127.0.0.1:36216/y_dge/g",
            "node_name":"127.0.0.1:36216_y_dge%2Fg",
            "state":"active",
            "type":"NRT",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"1",
    "tlogReplicas":"0"},
  "c8n_1x2":{
    "pullReplicas":"0",
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node3":{
            "core":"c8n_1x2_shard1_replica_t1",
            "base_url":"http://127.0.0.1:55265/y_dge/g",
            "node_name":"127.0.0.1:55265_y_dge%2Fg",
            "state":"active",
            "type":"TLOG",
            "leader":"true"},
          "core_node4":{
            "state":"down",
            "base_url":"http://127.0.0.1:59492/y_dge/g",
            "core":"c8n_1x2_shard1_replica_t2",
            "node_name":"127.0.0.1:59492_y_dge%2Fg",
            "type":"TLOG"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "nrtReplicas":"0",
    "tlogReplicas":"2"}}
        at __randomizedtesting.SeedInfo.seed([F5AA0A713AE9B538:7DFE35AB9415D8C0]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1992)
        at org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:367)
        at org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:131)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1487 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexSorting
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexSorting -Dtests.method=testRandom3 -Dtests.seed=DBC497A8C3B6A198 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ru-RU -Dtests.timezone=GB -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    525s J2 | TestIndexSorting.testRandom3 <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > at __randomizedtesting.SeedInfo.seed([DBC497A8C3B6A198:791CD972A744889E]:0)
   [junit4]    > at org.apache.lucene.util.packed.Packed8ThreeBlocks.<init>(Packed8ThreeBlocks.java:41)
   [junit4]    > at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:963)
   [junit4]    > at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:939)
   [junit4]    > at org.apache.lucene.util.packed.GrowableWriter.ensureCapacity(GrowableWriter.java:80)
   [junit4]    > at org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:88)
   [junit4]    > at org.apache.lucene.util.packed.AbstractPagedMutable.set(AbstractPagedMutable.java:98)
   [junit4]    > at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:152)
   [junit4]    > at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
   [junit4]    > at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
   [junit4]    > at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
   [junit4]    > at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:296)
   [junit4]    > at org.apache.lucene.util.fst.Builder.add(Builder.java:400)
   [junit4]    > at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.writeFST(MemoryDocValuesConsumer.java:373)
   [junit4]    > at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:416)
   [junit4]    > at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:406)
   [junit4]    > at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:527)
   [junit4]    > at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:139)
   [junit4]    > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:181)
   [junit4]    > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:125)
   [junit4]    > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4392)
   [junit4]    > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4032)
   [junit4]    > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]    > at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2235)
   [junit4]    > at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5050)
   [junit4]    > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1720)
   [junit4]    > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1462)
   [junit4]    > at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2301)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexSorting_DBC497A8C3B6A198-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {docs=PostingsFormat(name=Direct), sparse_text=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), norms=PostingsFormat(name=Direct), positions=PostingsFormat(name=Asserting), id=PostingsFormat(name=Direct), term_vectors=PostingsFormat(name=Direct)}, docValues:{multi_valued_long=DocValuesFormat(name=Asserting), foo=DocValuesFormat(name=Asserting), numeric=DocValuesFormat(name=Memory), float=DocValuesFormat(name=Direct), multi_valued_numeric=DocValuesFormat(name=Memory), long=DocValuesFormat(name=Direct), points=DocValuesFormat(name=Lucene70), sparse_int=DocValuesFormat(name=Direct), bar=DocValuesFormat(name=Memory), multi_valued_double=DocValuesFormat(name=Memory), docs=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Memory), multi_valued_int=DocValuesFormat(name=Lucene70), multi_valued_bytes=DocValuesFormat(name=Memory), sparse_text=DocValuesFormat(name=Direct), double=DocValuesFormat(name=Memory), positions=DocValuesFormat(name=Asserting), sparse_binary=DocValuesFormat(name=Direct), int=DocValuesFormat(name=Memory), sorted=DocValuesFormat(name=Memory), sparse=DocValuesFormat(name=Direct), multi_valued_string=DocValuesFormat(name=Memory), norms=DocValuesFormat(name=Memory), bytes=DocValuesFormat(name=Memory), binary=DocValuesFormat(name=Memory), dense_int=DocValuesFormat(name=Memory), multi_valued_float=DocValuesFormat(name=Asserting), term_vectors=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1490, maxMBSortInHeap=6.125264239773882, sim=RandomSimilarity(queryNorm=true): {positions=DFR GLZ(0.3), id=DFR I(ne)1, term_vectors=BM25(k1=1.2,b=0.75)}, locale=ru-RU, timezone=GB
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=357846920,total=406847488
   [junit4]   2> NOTE: All tests run in this JVM: [TestScorerPerf, TestMultiMMap, TestMultiLevelSkipList, TestTopDocsCollector, Test2BBinaryDocValues, TestRamUsageEstimator, TestRollingBuffer, TestHighCompressionMode, TestCollectionUtil, TestIntsRef, TestMultiset, TestPayloads, TestCharsRef, TestParallelCompositeReader, TestSimpleExplanationsOfNonMatches, TestSegmentTermEnum, TestSpanBoostQuery, TestConsistentFieldNumbers, Test2BNumericDocValues, TestBooleanSimilarity, TestNRTReaderWithThreads, TestIndexCommit, TestEarlyTerminatingSortingCollector, TestBoolean2ScorerSupplier, TestSameTokenSamePosition, TestNumericUtils, TestRadixSelector, TestDateSort, TestLongPostings, TestPrefixQuery, TestIndexWriterMergePolicy, TestAttributeSource, TestDemoParallelLeafReader, TestPackedInts, TestConjunctions, TestSloppyMath, TestStressAdvance, TestRollingUpdates, TestLSBRadixSorter, TestNoMergePolicy, TestSleepingLockWrapper, TestSpanTermQuery, TestSubScorerFreqs, Test2BPagedBytes, TestAllFilesHaveChecksumFooter, Test2BPositions, TestHugeRamFile, TestLazyProxSkipping, TestTermRangeQuery, TestConjunctionDISI, TestAssertions, TestPackedTokenAttributeImpl, TestNRTThreads, TestTopDocsMerge, TestMergePolicyWrapper, TestDisjunctionMaxQuery, TestSimilarityBase, TestCharTermAttributeImpl, TestFieldsReader, TestSetOnce, TestBlockPostingsFormat, Test2BSortedDocValuesOrds, TestFilterWeight, TestBagOfPostings, TestFixedLengthBytesRefArray, TestIndexWriterLockRelease, TestLucene60PointsFormat, TestAutomaton, TestByteBlockPool, TestIndexWriterThreadsToSegments, TestIOUtils, TestSloppyPhraseQuery2, TestBasics, TestLogMergePolicy, TestPolygon2D, TestFieldReuse, TestIndexWriterReader, TestIndexWriterMerging, TestIndexWriterWithThreads, TestShardSearching, TestDeletionPolicy, TestStandardAnalyzer, TestIndexWriterCommit, TestSearchWithThreads, TestConcurrentMergeScheduler, TestDocumentsWriterStallControl, TestDoc, TestSimilarity2, TestExternalCodecs, TestNoDeletionPolicy, TestRegexpQuery, TestPerSegmentDeletes, TestPrefixInBooleanQuery, TestComplexExplanationsOfNonMatches, TestAllFilesHaveCodecHeader, TestAutomatonQuery, TestDocument, TestTermScorer, TestTopFieldCollector, TestFileSwitchDirectory, TestSumDocFreq, TestDirectory, TestPayloadsOnVectors, TestBooleanQueryVisitSubscorers, TestSizeBoundedForceMerge, TestParallelTermEnum, TestBinaryTerms, TestMultiTermQueryRewrites, TestField, TestSpanExplanationsOfNonMatches, TestSimilarityProvider, TestCachingTokenFilter, TestDateTools, TestBooleanScorer, TestForUtil, TestLucene50CompoundFormat, TestLucene50TermVectorsFormat, TestLucene70SegmentInfoFormat, TestGeoEncodingUtils, TestGeoUtils, Test2BPoints, Test2BSortedDocValuesFixedSorted, TestBinaryDocValuesUpdates, TestCustomTermFreq, TestDocValues, TestDocsWithFieldSet, TestExceedMaxTermLength, TestExitableDirectoryReader, TestFieldInvertState, TestFilterDirectoryReader, TestIndexManyDocuments, TestIndexSorting]
   [junit4] Completed [373/454 (1!)] on J2 in 559.48s, 50 tests, 1 error <<< FAILURES!

[...truncated 272 lines...]
   [junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/lucene/build/core/test/temp/junit4-J2-20170725_233852_2494397812854820730776.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid13501.hprof ...
   [junit4] Heap dump file created [418647565 bytes in 1.365 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 10106 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/init-core-data-001
   [junit4]   2> 863006 WARN  (SUITE-HttpPartitionTest-seed#[F5AA0A713AE9B538]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 863013 INFO  (SUITE-HttpPartitionTest-seed#[F5AA0A713AE9B538]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 863015 INFO  (SUITE-HttpPartitionTest-seed#[F5AA0A713AE9B538]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 863015 INFO  (SUITE-HttpPartitionTest-seed#[F5AA0A713AE9B538]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 863015 INFO  (SUITE-HttpPartitionTest-seed#[F5AA0A713AE9B538]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /y_dge/g
   [junit4]   2> 863020 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 863020 INFO  (Thread-3481) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 863020 INFO  (Thread-3481) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 863024 ERROR (Thread-3481) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 863120 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.ZkTestServer start zk server on port:52242
   [junit4]   2> 863217 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 863218 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 863220 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 863221 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 863222 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 863223 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 863224 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 863225 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 863226 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 863227 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 863228 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 863230 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 863430 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 863433 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4961ca44{/y_dge/g,null,AVAILABLE}
   [junit4]   2> 863434 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@44205b57{HTTP/1.1,[http/1.1]}{127.0.0.1:41831}
   [junit4]   2> 863434 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.e.j.s.Server Started @866786ms
   [junit4]   2> 863434 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/tempDir-001/control/data, hostContext=/y_dge/g, hostPort=36216, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/control-001/cores}
   [junit4]   2> 863435 ERROR (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 863435 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 863435 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 863435 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 863435 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-26T03:20:57.295Z
   [junit4]   2> 863457 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 863457 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/control-001/solr.xml
   [junit4]   2> 863462 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5fba27d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 863466 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52242/solr
   [junit4]   2> 863710 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 863711 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36216_y_dge%2Fg
   [junit4]   2> 863712 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.Overseer Overseer (id=98372108739149828-127.0.0.1:36216_y_dge%2Fg-n_0000000000) starting
   [junit4]   2> 863724 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36216_y_dge%2Fg
   [junit4]   2> 863725 INFO  (zkCallback-1123-thread-1-processing-n:127.0.0.1:36216_y_dge%2Fg) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 863812 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fba27d
   [junit4]   2> 863818 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fba27d
   [junit4]   2> 863818 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fba27d
   [junit4]   2> 863819 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/control-001/cores
   [junit4]   2> 863858 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 863858 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:52242/solr ready
   [junit4]   2> 863859 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=49815,localport=36216], receiveBufferSize:531000
   [junit4]   2> 863867 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=41831,localport=48954], receiveBufferSize=530904
   [junit4]   2> 863868 INFO  (qtp187955345-9157) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:36216_y_dge%252Fg&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 863875 INFO  (OverseerThreadFactory-3028-thread-1-processing-n:127.0.0.1:36216_y_dge%2Fg) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 863987 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=49817,localport=36216], receiveBufferSize:531000
   [junit4]   2> 864000 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=41831,localport=48956], receiveBufferSize=530904
   [junit4]   2> 864000 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 864001 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 864111 INFO  (zkCallback-1123-thread-1-processing-n:127.0.0.1:36216_y_dge%2Fg) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 865025 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 865046 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 865165 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 865181 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 865181 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fba27d
   [junit4]   2> 865182 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 865182 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 865184 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@87296c1
   [junit4]   2> 865188 WARN  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 865252 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 865252 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 865254 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 865255 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 865256 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8316865634653194]
   [junit4]   2> 865256 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7c26a15e[control_collection_shard1_replica_n1] main]
   [junit4]   2> 865258 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 865258 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 865259 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 865260 INFO  (searcherExecutor-3031-thread-1-processing-n:127.0.0.1:36216_y_dge%2Fg x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@7c26a15e[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 865267 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1573953742178353152
   [junit4]   2> 865276 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 865276 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 865276 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:36216/y_dge/g/control_collection_shard1_replica_n1/
   [junit4]   2> 865276 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 865276 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:36216/y_dge/g/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 865276 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 865280 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:36216/y_dge/g/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 865407 INFO  (zkCallback-1123-thread-1-processing-n:127.0.0.1:36216_y_dge%2Fg) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 865447 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 865449 INFO  (qtp187955345-9159) [n:127.0.0.1:36216_y_dge%2Fg c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1448
   [junit4]   2> 865458 INFO  (qtp187955345-9157) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 865553 INFO  (zkCallback-1123-thread-3-processing-n:127.0.0.1:36216_y_dge%2Fg) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 865877 INFO  (OverseerCollectionConfigSetProcessor-98372108739149828-127.0.0.1:36216_y_dge%2Fg-n_0000000000) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 866458 INFO  (qtp187955345-9157) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:36216_y_dge%252Fg&wt=javabin&version=2} status=0 QTime=2590
   [junit4]   2> 866470 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 866471 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:52242/solr ready
   [junit4]   2> 866471 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 866472 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=49823,localport=36216], receiveBufferSize:531000
   [junit4]   2> 866472 INFO  (SocketProxy-Acceptor-36216) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=41831,localport=48962], receiveBufferSize=530904
   [junit4]   2> 866474 INFO  (qtp187955345-9155) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 866478 INFO  (OverseerThreadFactory-3028-thread-2-processing-n:127.0.0.1:36216_y_dge%2Fg) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 866479 WARN  (OverseerThreadFactory-3028-thread-2-processing-n:127.0.0.1:36216_y_dge%2Fg) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 866684 INFO  (qtp187955345-9155) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 866685 INFO  (qtp187955345-9155) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=210
   [junit4]   2> 866829 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/shard-1-001 of type TLOG
   [junit4]   2> 866829 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 866831 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7dde83c2{/y_dge/g,null,AVAILABLE}
   [junit4]   2> 866831 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4b3664ce{HTTP/1.1,[http/1.1]}{127.0.0.1:59953}
   [junit4]   2> 866831 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.e.j.s.Server Started @870183ms
   [junit4]   2> 866831 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/y_dge/g, hostPort=55265, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/shard-1-001/cores}
   [junit4]   2> 866838 ERROR (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 866838 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 866838 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 866838 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 866838 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-26T03:21:00.698Z
   [junit4]   2> 866844 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 866844 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_F5AA0A713AE9B538-001/shard-1-001/solr.xml
   [junit4]   2> 866858 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5fba27d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 866867 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52242/solr
   [junit4]   2> 866882 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:55265_y_dge%2Fg    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 866884 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:55265_y_dge%2Fg    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 866886 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:55265_y_dge%2Fg    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55265_y_dge%2Fg
   [junit4]   2> 866892 INFO  (zkCallback-1130-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 866904 INFO  (zkCallback-1123-thread-3-processing-n:127.0.0.1:36216_y_dge%2Fg) [n:127.0.0.1:36216_y_dge%2Fg    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 866906 INFO  (zkCallback-1135-thread-1-processing-n:127.0.0.1:55265_y_dge%2Fg) [n:127.0.0.1:55265_y_dge%2Fg    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 866960 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:55265_y_dge%2Fg    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fba27d
   [junit4]   2> 866966 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:55265_y_dge%2Fg    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fba27d
   [junit4]   2> 866966 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:55265_y_dge%2Fg    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fba27d
   [junit4]   2> 866968 INFO  (TEST-HttpPartitionTest.test-seed#[F5AA0A713AE9B538]) [n:127.0.0.1:55265_y_dge%2Fg    ] o.a.s.c.CoreProperties

[...truncated too long message...]

      at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
   [junit4]    >         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4]    >         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.flush(PrintStream.java:338)
   [junit4]    >         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.write(PrintStream.java:482)
   [junit4]    >         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
   [junit4]    >         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
   [junit4]    >         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
   [junit4]    >         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
   [junit4]    >         at java.io.PrintStream.write(PrintStream.java:527)
   [junit4]    >         at java.io.PrintStream.print(PrintStream.java:669)
   [junit4]    >         at java.io.PrintStream.println(PrintStream.java:806)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
   [junit4]    >    2) Thread[id=10492, name=zkCallback-1246-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    3) Thread[id=10491, name=zkCallback-1246-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=10490, name=zkCallback-1246-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    5) Thread[id=10382, name=zkCallback-1246-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    6) Thread[id=10380, name=StoppableCommitThread-SendThread(127.0.0.1:34821), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   [junit4]    >    7) Thread[id=10211, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at org.apache.solr.common.util.ByteUtils.UTF16toUTF8(ByteUtils.java:90)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:840)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:932)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
   [junit4]    >         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([F5AA0A713AE9B538]:0)
   [junit4] Completed [191/726 (2!)] on J1 in 111.02s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 9176 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:818: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:762: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid13501.hprof

Total time: 274 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 18 - Still unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/18/

7 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([524401EB206BCFA5:DA103E318E97A25D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:222)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
19 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:     1) Thread[id=23199, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)    2) Thread[id=23275, name=zkCallback-2320-thread-3, state=TERMINATED, group={null group}]         at (empty stack)    3) Thread[id=23123, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[524401EB206BCFA5]-SendThread(127.0.0.1:56569), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    4) Thread[id=23200, name=FullThrottleStopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.util.WeakHashMap.getTable(WeakHashMap.java:350)         at java.util.WeakHashMap.getEntry(WeakHashMap.java:427)         at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)         at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)         at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)         at org.apache.solr.cloud.FullThrottleStoppableIndexingThread.run(FullThrottleStoppableIndexingThread.java:76)    5) Thread[id=23201, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=23197, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    7) Thread[id=23124, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[524401EB206BCFA5]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    8) Thread[id=23249, name=StoppableCommitThread-SendThread(127.0.0.1:56569), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    9) Thread[id=23195, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)   10) Thread[id=23198, name=StoppableSearchThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:65)   11) Thread[id=23250, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)   12) Thread[id=23122, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)   13) Thread[id=23196, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 19 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:
   1) Thread[id=23199, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   2) Thread[id=23275, name=zkCallback-2320-thread-3, state=TERMINATED, group={null group}]
        at (empty stack)
   3) Thread[id=23123, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[524401EB206BCFA5]-SendThread(127.0.0.1:56569), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   4) Thread[id=23200, name=FullThrottleStopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.util.WeakHashMap.getTable(WeakHashMap.java:350)
        at java.util.WeakHashMap.getEntry(WeakHashMap.java:427)
        at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)
        at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
        at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
        at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)
        at org.apache.solr.cloud.FullThrottleStoppableIndexingThread.run(FullThrottleStoppableIndexingThread.java:76)
   5) Thread[id=23201, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=23197, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   7) Thread[id=23124, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[524401EB206BCFA5]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   8) Thread[id=23249, name=StoppableCommitThread-SendThread(127.0.0.1:56569), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   9) Thread[id=23195, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
  10) Thread[id=23198, name=StoppableSearchThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:65)
  11) Thread[id=23250, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
  12) Thread[id=23122, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
  13) Thread[id=23196, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
        at __randomizedtesting.SeedInfo.seed([524401EB206BCFA5]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=23195, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    2) Thread[id=23198, name=StoppableSearchThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at org.apache.solr.logging.LogWatcher.add(LogWatcher.java:70)         at org.apache.solr.logging.log4j.EventAppender.append(EventAppender.java:36)         at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)         at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)         at org.apache.log4j.Category.callAppenders(Category.java:206)         at org.apache.log4j.Category.forcedLog(Category.java:391)         at org.apache.log4j.Category.log(Category.java:856)         at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:540)         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:897)         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)         at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)         at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:54)    3) Thread[id=23123, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[524401EB206BCFA5]-SendThread(127.0.0.1:56569), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    4) Thread[id=23200, name=FullThrottleStopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Object.hashCode(Native Method)         at java.util.WeakHashMap.hash(WeakHashMap.java:298)         at java.util.WeakHashMap.get(WeakHashMap.java:396)         at com.carrotsearch.randomizedtesting.RandomizedContext.getPerThread(RandomizedContext.java:319)         at com.carrotsearch.randomizedtesting.RandomizedContext.getRandomness(RandomizedContext.java:92)         at com.carrotsearch.randomizedtesting.RandomizedContext.getRandom(RandomizedContext.java:114)         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)         at org.apache.solr.cloud.FullThrottleStoppableIndexingThread.run(FullThrottleStoppableIndexingThread.java:76)    5) Thread[id=23197, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.io.PrintStream.println(PrintStream.java:805)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)    6) Thread[id=23196, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)    7) Thread[id=23249, name=StoppableCommitThread-SendThread(127.0.0.1:56569), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=23195, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   2) Thread[id=23198, name=StoppableSearchThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at org.apache.solr.logging.LogWatcher.add(LogWatcher.java:70)
        at org.apache.solr.logging.log4j.EventAppender.append(EventAppender.java:36)
        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
        at org.apache.log4j.Category.callAppenders(Category.java:206)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:540)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:897)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:54)
   3) Thread[id=23123, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[524401EB206BCFA5]-SendThread(127.0.0.1:56569), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   4) Thread[id=23200, name=FullThrottleStopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Object.hashCode(Native Method)
        at java.util.WeakHashMap.hash(WeakHashMap.java:298)
        at java.util.WeakHashMap.get(WeakHashMap.java:396)
        at com.carrotsearch.randomizedtesting.RandomizedContext.getPerThread(RandomizedContext.java:319)
        at com.carrotsearch.randomizedtesting.RandomizedContext.getRandomness(RandomizedContext.java:92)
        at com.carrotsearch.randomizedtesting.RandomizedContext.getRandom(RandomizedContext.java:114)
        at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)
        at org.apache.solr.cloud.FullThrottleStoppableIndexingThread.run(FullThrottleStoppableIndexingThread.java:76)
   5) Thread[id=23197, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.io.PrintStream.println(PrintStream.java:805)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
   6) Thread[id=23196, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
        at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
        at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
        at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
        at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   7) Thread[id=23249, name=StoppableCommitThread-SendThread(127.0.0.1:56569), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
        at __randomizedtesting.SeedInfo.seed([524401EB206BCFA5]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([524401EB206BCFA5:DA103E318E97A25D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
6 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) Thread[id=16917, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)    2) Thread[id=16970, name=zkCallback-2421-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=16951, name=zkCallback-2421-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=16950, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    5) Thread[id=16971, name=zkCallback-2421-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=16949, name=StoppableCommitThread-SendThread(127.0.0.1:46935), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
   1) Thread[id=16917, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   2) Thread[id=16970, name=zkCallback-2421-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=16951, name=zkCallback-2421-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=16950, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   5) Thread[id=16971, name=zkCallback-2421-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=16949, name=StoppableCommitThread-SendThread(127.0.0.1:46935), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
        at __randomizedtesting.SeedInfo.seed([524401EB206BCFA5]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16970, name=zkCallback-2421-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=16951, name=zkCallback-2421-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=16971, name=zkCallback-2421-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=16949, name=StoppableCommitThread-SendThread(127.0.0.1:46935), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16970, name=zkCallback-2421-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=16951, name=zkCallback-2421-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=16971, name=zkCallback-2421-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=16949, name=StoppableCommitThread-SendThread(127.0.0.1:46935), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
        at __randomizedtesting.SeedInfo.seed([524401EB206BCFA5]:0)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
expected:<2> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<3>
        at __randomizedtesting.SeedInfo.seed([524401EB206BCFA5:DA103E318E97A25D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.assertSliceAndReplicaCount(SharedFSAutoReplicaFailoverTest.java:403)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:248)
        at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12377 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_524401EB206BCFA5-001/init-core-data-001
   [junit4]   2> 1672035 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 1672038 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1672040 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 1672040 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1672041 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1672081 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1672088 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1672090 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1672110 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_40950_hdfs____.q2fd1a/webapp
   [junit4]   2> 1672538 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40950
   [junit4]   2> 1672666 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1672667 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1672684 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_49089_datanode____.vcif0t/webapp
   [junit4]   2> 1673397 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:49089
   [junit4]   2> 1673498 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1673507 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1673533 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_53483_datanode____sh2uk8/webapp
   [junit4]   2> 1673630 INFO  (IPC Server handler 5 on 37979) [    ] BlockStateChange BLOCK* processReport: from storage DS-e0a1202b-6082-4df6-9d67-bf203a382596 node DatanodeRegistration(127.0.0.1:33205, datanodeUuid=799b5afb-f1cd-4b40-9124-faa1256dfa15, infoPort=35275, infoSecurePort=0, ipcPort=44590, storageInfo=lv=-56;cid=testClusterID;nsid=613817770;c=0), blocks: 0, hasStaleStorage: true, processing time: 3 msecs
   [junit4]   2> 1673630 INFO  (IPC Server handler 5 on 37979) [    ] BlockStateChange BLOCK* processReport: from storage DS-40efbbf6-076b-401a-9d28-cda099c06f6e node DatanodeRegistration(127.0.0.1:33205, datanodeUuid=799b5afb-f1cd-4b40-9124-faa1256dfa15, infoPort=35275, infoSecurePort=0, ipcPort=44590, storageInfo=lv=-56;cid=testClusterID;nsid=613817770;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1674060 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[524401EB206BCFA5]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53483
   [junit4]   2> 1674236 INFO  (IPC Server handler 8 on 37979) [    ] BlockStateChange BLOCK* processReport: from storage DS-da7ceb54-3c87-463a-8af2-d6bd8daa4215 node DatanodeRegistration(127.0.0.1:43207, datanodeUuid=01ddc550-003e-4ef3-ac1a-12629f5e87da, infoPort=59535, infoSecurePort=0, ipcPort=54106, storageInfo=lv=-56;cid=testClusterID;nsid=613817770;c=0), blocks: 0, hasStaleStorage: true, processing time: 3 msecs
   [junit4]   2> 1674236 INFO  (IPC Server handler 8 on 37979) [    ] BlockStateChange BLOCK* processReport: from storage DS-50199f66-a532-45e2-88b7-aba159090786 node DatanodeRegistration(127.0.0.1:43207, datanodeUuid=01ddc550-003e-4ef3-ac1a-12629f5e87da, infoPort=59535, infoSecurePort=0, ipcPort=54106, storageInfo=lv=-56;cid=testClusterID;nsid=613817770;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1674583 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1674584 INFO  (Thread-7847) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1674584 INFO  (Thread-7847) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1674589 ERROR (Thread-7847) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1674684 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.ZkTestServer start zk server on port:59165
   [junit4]   2> 1674706 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15d82981b320000, likely client has closed socket
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1674722 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1674724 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1674726 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1674728 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1674729 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1674731 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1674732 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1674734 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1674736 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1674737 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1674739 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1674741 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1674892 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1674894 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7a175c9d{/,null,AVAILABLE}
   [junit4]   2> 1674894 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3f47afac{HTTP/1.1,[http/1.1]}{127.0.0.1:39354}
   [junit4]   2> 1674897 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.e.j.s.Server Started @1677807ms
   [junit4]   2> 1674897 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=39354, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_524401EB206BCFA5-001/control-001/cores}
   [junit4]   2> 1674898 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1674898 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 1674898 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1674898 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1674898 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-27T05:49:53.131Z
   [junit4]   2> 1674904 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1674904 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_524401EB206BCFA5-001/control-001/solr.xml
   [junit4]   2> 1674911 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@315e8932, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1674916 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59165/solr
   [junit4]   2> 1675033 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [n:127.0.0.1:39354_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1675034 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [n:127.0.0.1:39354_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39354_
   [junit4]   2> 1675035 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [n:127.0.0.1:39354_    ] o.a.s.c.Overseer Overseer (id=98378356674920452-127.0.0.1:39354_-n_0000000000) starting
   [junit4]   2> 1675047 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [n:127.0.0.1:39354_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39354_
   [junit4]   2> 1675049 INFO  (zkCallback-1059-thread-1-processing-n:127.0.0.1:39354_) [n:127.0.0.1:39354_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1675165 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [n:127.0.0.1:39354_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@315e8932
   [junit4]   2> 1675176 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [n:127.0.0.1:39354_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@315e8932
   [junit4]   2> 1675176 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [n:127.0.0.1:39354_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@315e8932
   [junit4]   2> 1675178 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [n:127.0.0.1:39354_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_524401EB206BCFA5-001/control-001/cores
   [junit4]   2> 1675201 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1675203 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59165/solr ready
   [junit4]   2> 1675204 INFO  (qtp1955218730-14843) [n:127.0.0.1:39354_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:39354_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1675210 INFO  (OverseerThreadFactory-3242-thread-1-processing-n:127.0.0.1:39354_) [n:127.0.0.1:39354_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1675318 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1675318 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1675424 INFO  (zkCallback-1059-thread-1-processing-n:127.0.0.1:39354_) [n:127.0.0.1:39354_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1676342 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 1676358 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1676468 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1676498 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1676498 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@315e8932
   [junit4]   2> 1676498 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:37979/solr_hdfs_home
   [junit4]   2> 1676499 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1676499 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1676499 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_524401EB206BCFA5-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://localhost:37979/solr_hdfs_home/control_collection/core_node2/data/]
   [junit4]   2> 1676501 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:37979/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 1676510 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1676511 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 1676535 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1676545 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:37979/solr_hdfs_home/control_collection/core_node2/data
   [junit4]   2> 1676571 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:37979/solr_hdfs_home/control_collection/core_node2/data/index
   [junit4]   2> 1676579 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1676579 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 1676597 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1676598 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=50, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7914392041261432]
   [junit4]   2> 1676684 INFO  (IPC Server handler 5 on 37979) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43207 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-e0a1202b-6082-4df6-9d67-bf203a382596:NORMAL:127.0.0.1:33205|RBW], ReplicaUC[[DISK]DS-50199f66-a532-45e2-88b7-aba159090786:NORMAL:127.0.0.1:43207|RBW]]} size 0
   [junit4]   2> 1676704 INFO  (IPC Server handler 9 on 37979) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33205 is added to blk_1073741825_1001 size 72
   [junit4]   2> 1676715 WARN  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1676789 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1676789 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1676789 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 1676803 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1676803 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1676806 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1676814 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@37a68d5a[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1676815 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1676816 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1676817 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1676818 INFO  (searcherExecutor-3245-thread-1-processing-n:127.0.0.1:39354_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@37a68d5a[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1676826 WARN  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.h.HdfsLocalityReporter Could not retrieve locality information for hdfs://localhost:45436/solr3 due to exception: java.net.ConnectException: Call From lucene1-us-west/10.41.0.5 to localhost:45436 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
   [junit4]   2> 1676826 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1574053709148585984
   [junit4]   2> 1676834 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1676834 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1676834 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:39354/control_collection_shard1_replica_n1/
   [junit4]   2> 1676834 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1676834 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:39354/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1676835 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1676839 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:39354/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1676942 INFO  (zkCallback-1059-thread-1-processing-n:127.0.0.1:39354_) [n:127.0.0.1:39354_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1676990 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1676992 INFO  (qtp1955218730-14845) [n:127.0.0.1:39354_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1674
   [junit4]   2> 1677001 INFO  (qtp1955218730-14843) [n:127.0.0.1:39354_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1677099 INFO  (zkCallback-1059-thread-1-processing-n:127.0.0.1:39354_) [n:127.0.0.1:39354_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1677214 INFO  (OverseerCollectionConfigSetProcessor-98378356674920452-127.0.0.1:39354_-n_0000000000) [n:127.0.0.1:39354_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1678002 INFO  (qtp1955218730-14843) [n:127.0.0.1:39354_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:39354_&wt=javabin&version=2} status=0 QTime=2797
   [junit4]   2> 1678009 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1678010 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59165/solr ready
   [junit4]   2> 1678010 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1678012 INFO  (qtp1955218730-14841) [n:127.0.0.1:39354_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1678014 INFO  (OverseerThreadFactory-3242-thread-2-processing-n:127.0.0.1:39354_) [n:127.0.0.1:39354_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 1678015 WARN  (OverseerThreadFactory-3242-thread-2-processing-n:127.0.0.1:39354_) [n:127.0.0.1:39354_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 1678225 INFO  (qtp1955218730-14841) [n:127.0.0.1:39354_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1678225 INFO  (qtp1955218730-14841) [n:127.0.0.1:39354_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=213
   [junit4]   2> 1678392 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_524401EB206BCFA5-001/shard-1-001 of type NRT
   [junit4]   2> 1678393 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1678394 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1ff76c0b{/,null,AVAILABLE}
   [junit4]   2> 1678395 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@10ed4fb1{HTTP/1.1,[http/1.1]}{127.0.0.1:54386}
   [junit4]   2> 1678395 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.e.j.s.Server Started @1681305ms
   [junit4]   2> 1678395 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solrconfig=solrconfig.xml, hostContext=/, hostPort=54386, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_524401EB206BCFA5-001/shard-1-001/cores}
   [junit4]   2> 1678398 ERROR (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1678401 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.1.0
   [junit4]   2> 1678401 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[524401EB206BCFA5]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]  

[...truncated too long message...]

MED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >   13) Thread[id=23196, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([524401EB206BCFA5]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=23195, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
   [junit4]    >         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]    >    2) Thread[id=23198, name=StoppableSearchThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at org.apache.solr.logging.LogWatcher.add(LogWatcher.java:70)
   [junit4]    >         at org.apache.solr.logging.log4j.EventAppender.append(EventAppender.java:36)
   [junit4]    >         at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
   [junit4]    >         at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
   [junit4]    >         at org.apache.log4j.Category.callAppenders(Category.java:206)
   [junit4]    >         at org.apache.log4j.Category.forcedLog(Category.java:391)
   [junit4]    >         at org.apache.log4j.Category.log(Category.java:856)
   [junit4]    >         at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:540)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:897)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
   [junit4]    >         at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
   [junit4]    >         at org.apache.solr.cloud.StoppableSearchThread.run(StoppableSearchThread.java:54)
   [junit4]    >    3) Thread[id=23123, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[524401EB206BCFA5]-SendThread(127.0.0.1:56569), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   [junit4]    >    4) Thread[id=23200, name=FullThrottleStopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at java.lang.Object.hashCode(Native Method)
   [junit4]    >         at java.util.WeakHashMap.hash(WeakHashMap.java:298)
   [junit4]    >         at java.util.WeakHashMap.get(WeakHashMap.java:396)
   [junit4]    >         at com.carrotsearch.randomizedtesting.RandomizedContext.getPerThread(RandomizedContext.java:319)
   [junit4]    >         at com.carrotsearch.randomizedtesting.RandomizedContext.getRandomness(RandomizedContext.java:92)
   [junit4]    >         at com.carrotsearch.randomizedtesting.RandomizedContext.getRandom(RandomizedContext.java:114)
   [junit4]    >         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)
   [junit4]    >         at org.apache.solr.cloud.FullThrottleStoppableIndexingThread.run(FullThrottleStoppableIndexingThread.java:76)
   [junit4]    >    5) Thread[id=23197, name=StoppableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at java.io.PrintStream.println(PrintStream.java:805)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:120)
   [junit4]    >    6) Thread[id=23196, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:612)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228)
   [junit4]    >         at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155)
   [junit4]    >         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91)
   [junit4]    >         at org.apache.solr.client.solrj.impl.BinaryRequestWriter.getContentStream(BinaryRequestWriter.java:67)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
   [junit4]    >         at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:381)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:176)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115)
   [junit4]    >    7) Thread[id=23249, name=StoppableCommitThread-SendThread(127.0.0.1:56569), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([524401EB206BCFA5]:0)
   [junit4] Completed [544/726 (3!)] on J1 in 167.54s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 43818 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 19 - Still Unstable

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/19/

4 tests failed.
FAILED:  org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
        at __randomizedtesting.SeedInfo.seed([4624F69347029D00:6A16811ECBC6EB84]:0)
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:886)
        at org.apache.solr.update.processor.AtomicUpdateProcessorFactoryTest.testMultipleThreads(AtomicUpdateProcessorFactoryTest.java:264)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result[@numFound=1]
        xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

        request was:q=int_i:14&wt=xml
        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:879)
        ... 40 more


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties were stopped - this is worth investigating!
        at __randomizedtesting.SeedInfo.seed([4624F69347029D00:CE70C949E9FEF0F8]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
7 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) Thread[id=5847, name=zkCallback-916-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=5845, name=StoppableCommitThread-SendThread(127.0.0.1:33883), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    3) Thread[id=5821, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)    4) Thread[id=5888, name=zkCallback-916-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=5846, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    6) Thread[id=5886, name=zkCallback-916-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=5887, name=zkCallback-916-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 7 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
   1) Thread[id=5847, name=zkCallback-916-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=5845, name=StoppableCommitThread-SendThread(127.0.0.1:33883), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   3) Thread[id=5821, name=StoppableCommitThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   4) Thread[id=5888, name=zkCallback-916-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=5846, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   6) Thread[id=5886, name=zkCallback-916-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=5887, name=zkCallback-916-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([4624F69347029D00]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=5847, name=zkCallback-916-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=5845, name=StoppableCommitThread-SendThread(127.0.0.1:33883), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    3) Thread[id=5888, name=zkCallback-916-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=5886, name=zkCallback-916-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=5887, name=zkCallback-916-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=5847, name=zkCallback-916-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=5845, name=StoppableCommitThread-SendThread(127.0.0.1:33883), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   3) Thread[id=5888, name=zkCallback-916-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=5886, name=zkCallback-916-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=5887, name=zkCallback-916-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([4624F69347029D00]:0)




Build Log:
[...truncated 11959 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest
   [junit4]   2> 914147 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[4624F69347029D00]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/init-core-data-001
   [junit4]   2> 914147 WARN  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[4624F69347029D00]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 914147 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[4624F69347029D00]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 914177 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[4624F69347029D00]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 914178 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[4624F69347029D00]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /yn/
   [junit4]   2> 914194 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.ChaosMonkeySafeLeaderWithPullReplicasTest Starting ChaosMonkey test with 1 shards and 4 nodes
   [junit4]   2> 914229 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 914242 INFO  (Thread-1889) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 914242 INFO  (Thread-1889) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 914294 ERROR (Thread-1889) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 914342 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.ZkTestServer start zk server on port:33883
   [junit4]   2> 914386 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 914386 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 914387 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 914400 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 914401 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 914401 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 914402 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 914403 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 914403 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 914404 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 914433 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 914511 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
   [junit4]   2> 914848 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 914872 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4d6b85f1{/yn,null,AVAILABLE}
   [junit4]   2> 914873 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@12d8b7df{SSL,[ssl, http/1.1]}{127.0.0.1:38475}
   [junit4]   2> 914873 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.Server Started @924190ms
   [junit4]   2> 914873 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/tempDir-001/control/data, replicaType=NRT, hostContext=/yn, hostPort=38475, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/control-001/cores}
   [junit4]   2> 914874 ERROR (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 914874 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 914874 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 914874 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 914874 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-28T07:00:48.424Z
   [junit4]   2> 914887 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 914887 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/control-001/solr.xml
   [junit4]   2> 914891 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 914929 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33883/solr
   [junit4]   2> 915359 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:38475_yn    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 915368 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:38475_yn    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38475_yn
   [junit4]   2> 915369 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:38475_yn    ] o.a.s.c.Overseer Overseer (id=98384297840345093-127.0.0.1:38475_yn-n_0000000000) starting
   [junit4]   2> 915429 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:38475_yn    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38475_yn
   [junit4]   2> 915444 INFO  (OverseerStateUpdate-98384297840345093-127.0.0.1:38475_yn-n_0000000000) [n:127.0.0.1:38475_yn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 915724 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:38475_yn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 915740 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:38475_yn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 915740 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:38475_yn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 915741 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:38475_yn    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/control-001/cores
   [junit4]   2> 915817 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 915817 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33883/solr ready
   [junit4]   2> 915970 INFO  (qtp369723330-5645) [n:127.0.0.1:38475_yn    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38475_yn&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 915971 INFO  (OverseerThreadFactory-1639-thread-1-processing-n:127.0.0.1:38475_yn) [n:127.0.0.1:38475_yn    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 916101 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 916101 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 916223 INFO  (zkCallback-881-thread-1-processing-n:127.0.0.1:38475_yn) [n:127.0.0.1:38475_yn    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 917146 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 917188 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 917516 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 917577 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 917577 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 917578 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 917578 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 917580 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=13.4609375, floorSegmentMB=2.13671875, forceMergeDeletesPctAllowed=26.282983652184377, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 917615 WARN  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 917889 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 917889 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 917890 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 917890 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 917908 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 917909 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@14a44b41[control_collection_shard1_replica_n1] main]
   [junit4]   2> 917910 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 917910 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 917910 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 917924 INFO  (searcherExecutor-1642-thread-1-processing-n:127.0.0.1:38475_yn x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@14a44b41[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 917933 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1574148769299038208
   [junit4]   2> 917947 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 917947 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 917947 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38475/yn/control_collection_shard1_replica_n1/
   [junit4]   2> 917947 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 917947 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:38475/yn/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 917947 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 917963 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:38475/yn/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 918095 INFO  (zkCallback-881-thread-1-processing-n:127.0.0.1:38475_yn) [n:127.0.0.1:38475_yn    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 918114 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 918121 INFO  (qtp369723330-5640) [n:127.0.0.1:38475_yn c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2020
   [junit4]   2> 918156 INFO  (qtp369723330-5645) [n:127.0.0.1:38475_yn    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 918269 INFO  (zkCallback-881-thread-3-processing-n:127.0.0.1:38475_yn) [n:127.0.0.1:38475_yn    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 919157 INFO  (qtp369723330-5645) [n:127.0.0.1:38475_yn    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38475_yn&wt=javabin&version=2} status=0 QTime=3187
   [junit4]   2> 919161 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 919162 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33883/solr ready
   [junit4]   2> 919162 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 919164 INFO  (qtp369723330-5647) [n:127.0.0.1:38475_yn    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 919166 INFO  (OverseerThreadFactory-1639-thread-2-processing-n:127.0.0.1:38475_yn) [n:127.0.0.1:38475_yn    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 919166 INFO  (OverseerCollectionConfigSetProcessor-98384297840345093-127.0.0.1:38475_yn-n_0000000000) [n:127.0.0.1:38475_yn    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 919167 WARN  (OverseerThreadFactory-1639-thread-2-processing-n:127.0.0.1:38475_yn) [n:127.0.0.1:38475_yn    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 919412 INFO  (qtp369723330-5647) [n:127.0.0.1:38475_yn    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 919412 INFO  (qtp369723330-5647) [n:127.0.0.1:38475_yn    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=248
   [junit4]   2> 919567 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-1-001 of type TLOG
   [junit4]   2> 919568 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 919569 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@e41ea05{/yn,null,AVAILABLE}
   [junit4]   2> 919570 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@406fc4a3{SSL,[ssl, http/1.1]}{127.0.0.1:36776}
   [junit4]   2> 919570 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.Server Started @928886ms
   [junit4]   2> 919570 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/yn, hostPort=36776, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-1-001/cores}
   [junit4]   2> 919570 ERROR (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 919570 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 919570 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 919571 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 919571 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-28T07:00:53.121Z
   [junit4]   2> 919573 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 919573 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-1-001/solr.xml
   [junit4]   2> 919577 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 919579 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33883/solr
   [junit4]   2> 919593 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:36776_yn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 919595 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:36776_yn    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 919596 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:36776_yn    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36776_yn
   [junit4]   2> 919597 INFO  (zkCallback-881-thread-3-processing-n:127.0.0.1:38475_yn) [n:127.0.0.1:38475_yn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 919597 INFO  (zkCallback-888-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 919597 INFO  (zkCallback-893-thread-1-processing-n:127.0.0.1:36776_yn) [n:127.0.0.1:36776_yn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 919696 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:36776_yn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 919713 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:36776_yn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 919713 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:36776_yn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 919714 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:36776_yn    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-1-001/cores
   [junit4]   2> 919826 INFO  (qtp553695369-5689) [n:127.0.0.1:36776_yn    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:36776_yn&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 919828 INFO  (OverseerThreadFactory-1639-thread-3-processing-n:127.0.0.1:38475_yn) [n:127.0.0.1:38475_yn    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:36776_yn for creating new replica
   [junit4]   2> 919828 INFO  (OverseerCollectionConfigSetProcessor-98384297840345093-127.0.0.1:38475_yn-n_0000000000) [n:127.0.0.1:38475_yn    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 919833 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t1&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 919833 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 921071 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 921169 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.s.IndexSchema [collection1_shard1_replica_t1] Schema name=test
   [junit4]   2> 921582 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 921634 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t1' using configuration from collection collection1, trusted=true
   [junit4]   2> 921634 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t1' (registry 'solr.core.collection1.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 921634 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 921634 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [[collection1_shard1_replica_t1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-1-001/cores/collection1_shard1_replica_t1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-1-001/cores/collection1_shard1_replica_t1/data/]
   [junit4]   2> 921650 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=13.4609375, floorSegmentMB=2.13671875, forceMergeDeletesPctAllowed=26.282983652184377, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 921673 WARN  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 921796 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 921796 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 921797 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 921797 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 921798 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 921812 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@38952419[collection1_shard1_replica_t1] main]
   [junit4]   2> 921812 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 921813 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 921813 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 921814 INFO  (searcherExecutor-1653-thread-1-processing-n:127.0.0.1:36776_yn x:collection1_shard1_replica_t1 s:shard1 c:collection1) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [collection1_shard1_replica_t1] Registered new searcher Searcher@38952419[collection1_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 921814 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1574148773368561664
   [junit4]   2> 921841 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 921841 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 921841 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:36776/yn/collection1_shard1_replica_t1/
   [junit4]   2> 921841 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 921841 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.SyncStrategy https://127.0.0.1:36776/yn/collection1_shard1_replica_t1/ has no replicas
   [junit4]   2> 921841 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 921842 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.ZkController collection1_shard1_replica_t1 stopping background replication from leader
   [junit4]   2> 921856 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:36776/yn/collection1_shard1_replica_t1/ shard1
   [junit4]   2> 922007 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 922018 INFO  (qtp553695369-5686) [n:127.0.0.1:36776_yn c:collection1 s:shard1  x:collection1_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t1&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2185
   [junit4]   2> 922020 INFO  (qtp553695369-5689) [n:127.0.0.1:36776_yn    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:36776_yn&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2} status=0 QTime=2193
   [junit4]   2> 922441 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-2-001 of type TLOG
   [junit4]   2> 922457 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 922458 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@12d230e4{/yn,null,AVAILABLE}
   [junit4]   2> 922458 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5c3c7cbc{SSL,[ssl, http/1.1]}{127.0.0.1:35520}
   [junit4]   2> 922458 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.e.j.s.Server Started @931775ms
   [junit4]   2> 922458 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/tempDir-001/jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/yn, hostPort=35520, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-2-001/cores}
   [junit4]   2> 922458 ERROR (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 922459 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.1.0
   [junit4]   2> 922459 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 922459 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 922459 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-07-28T07:00:56.009Z
   [junit4]   2> 922497 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 922497 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-2-001/solr.xml
   [junit4]   2> 922510 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 922512 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33883/solr
   [junit4]   2> 922530 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:35520_yn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 922532 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:35520_yn    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 922533 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:35520_yn    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35520_yn
   [junit4]   2> 922534 INFO  (zkCallback-888-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 922598 INFO  (zkCallback-893-thread-1-processing-n:127.0.0.1:36776_yn) [n:127.0.0.1:36776_yn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 922602 INFO  (zkCallback-881-thread-2-processing-n:127.0.0.1:38475_yn) [n:127.0.0.1:38475_yn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 922639 INFO  (zkCallback-899-thread-1-processing-n:127.0.0.1:35520_yn) [n:127.0.0.1:35520_yn    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 923079 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:35520_yn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 923096 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:35520_yn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 923096 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:35520_yn    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 923097 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[4624F69347029D00]) [n:127.0.0.1:35520_yn    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-2-001/cores
   [junit4]   2> 923186 INFO  (qtp553695369-5690) [n:127.0.0.1:36776_yn    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:35520_yn&action=ADDREPLICA&collection=collection1&shard=shard1&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 923196 INFO  (OverseerThreadFactory-1639-thread-4-processing-n:127.0.0.1:38475_yn) [n:127.0.0.1:38475_yn    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:35520_yn for creating new replica
   [junit4]   2> 923196 INFO  (OverseerCollectionConfigSetProcessor-98384297840345093-127.0.0.1:38475_yn-n_0000000000) [n:127.0.0.1:38475_yn    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 923290 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t3&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 923290 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 924390 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 924417 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.s.IndexSchema [collection1_shard1_replica_t3] Schema name=test
   [junit4]   2> 924683 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 924717 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t3' using configuration from collection collection1, trusted=true
   [junit4]   2> 924717 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_t3' (registry 'solr.core.collection1.shard1.replica_t3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@36b27452
   [junit4]   2> 924718 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 924718 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.c.SolrCore [[collection1_shard1_replica_t3] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-2-001/cores/collection1_shard1_replica_t3], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_4624F69347029D00-001/shard-2-001/cores/collection1_shard1_replica_t3/data/]
   [junit4]   2> 924733 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=13.4609375, floorSegmentMB=2.13671875, forceMergeDeletesPctAllowed=26.282983652184377, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 924775 WARN  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 924982 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 924982 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 924983 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
   [junit4]   2> 924983 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 924984 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 924985 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.s.SolrIndexSearcher Opening [Searcher@7a5a40a6[collection1_shard1_replica_t3] main]
   [junit4]   2> 924986 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 924986 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 924987 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 924988 INFO  (searcherExecutor-1664-thread-1-processing-n:127.0.0.1:35520_yn x:collection1_shard1_replica_t3 s:shard1 c:collection1) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.c.SolrCore [collection1_shard1_replica_t3] Registered new searcher Searcher@7a5a40a6[collection1_shard1_replica_t3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 924988 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1574148776696741888
   [junit4]   2> 924991 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.c.ZkController Core needs to recover:collection1_shard1_replica_t3
   [junit4]   2> 924991 INFO  (updateExecutor-896-thread-1-processing-n:127.0.0.1:35520_yn x:collection1_shard1_replica_t3 s:shard1 c:collection1) [n:127.0.0.1:35520_yn c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_t3] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 924991 INFO  (qtp1701182163-5718) [n:127.0.0.1:35520_yn c:collection1 s:shard1  x:collection1_shard1_replica_t3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_t3&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 Q