Quantcast

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3830 - Unstable!

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3830 - Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3830/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

4 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter

Error Message:
Collection not found: withShardField

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: withShardField
        at __randomizedtesting.SeedInfo.seed([3332E6EB014EF6B0:66620E79ADB73940]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1376)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1072)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:232)
        at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.PeerSyncReplicationTest.test

Error Message:
timeout waiting to see all nodes active

Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
        at __randomizedtesting.SeedInfo.seed([3332E6EB014EF6B0:BB66D931AFB29B48]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
        at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
        at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
        at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestReplicaProperties.test

Error Message:
timeout waiting for collection in cluster state: collection=testcollection

Stack Trace:
java.lang.RuntimeException: timeout waiting for collection in cluster state: collection=testcollection
        at __randomizedtesting.SeedInfo.seed([3332E6EB014EF6B0:BB66D931AFB29B48]:0)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForCollection(AbstractFullDistribZkTestBase.java:346)
        at org.apache.solr.cloud.TestReplicaProperties.test(TestReplicaProperties.java:63)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds

Error Message:
1: soft wasn't fast enough

Stack Trace:
java.lang.AssertionError: 1: soft wasn't fast enough
        at __randomizedtesting.SeedInfo.seed([3332E6EB014EF6B0:6F2748D2EACCB7C8]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds(SoftAutoCommitTest.java:358)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11271 lines...]
   [junit4] Suite: org.apache.solr.update.SoftAutoCommitTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.update.SoftAutoCommitTest_3332E6EB014EF6B0-001/init-core-data-001
   [junit4]   2> 615564 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 615569 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 615570 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 615570 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/lib, /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 615611 WARN  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 615612 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 615636 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 615700 WARN  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.s.IndexSchema [null] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 615702 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 615760 WARN  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 615762 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7de8f8a9
   [junit4]   2> 615763 WARN  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 615765 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7de8f8a9
   [junit4]   2> 615765 WARN  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 615765 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7de8f8a9
   [junit4]   2> 615767 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/lib, /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 615807 WARN  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 615809 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 615827 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 615890 WARN  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 615892 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 615902 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1
   [junit4]   2> 615902 WARN  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 615902 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7de8f8a9
   [junit4]   2> 615902 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.update.SoftAutoCommitTest_3332E6EB014EF6B0-001/init-core-data-001/]
   [junit4]   2> 615903 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7de8f8a9
   [junit4]   2> 615904 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=72.0673828125, floorSegmentMB=1.4677734375, forceMergeDeletesPctAllowed=7.790960966524666, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2677513751027836
   [junit4]   2> 615981 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 615981 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 615982 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 615982 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 615983 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=21.8291015625, floorSegmentMB=0.853515625, forceMergeDeletesPctAllowed=2.6137780804190704, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8142636797647553
   [junit4]   2> 615984 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@62bf249[collection1] main]
   [junit4]   2> 615984 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 615991 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 616035 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 616035 INFO  (searcherExecutor-1934-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 616036 INFO  (searcherExecutor-1934-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 616036 INFO  (searcherExecutor-1934-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 616036 INFO  (searcherExecutor-1934-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 616036 INFO  (searcherExecutor-1934-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 616036 INFO  (coreLoadExecutor-1933-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559264770387869696
   [junit4]   2> 616036 INFO  (searcherExecutor-1934-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 616037 INFO  (searcherExecutor-1934-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 616037 INFO  (searcherExecutor-1934-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 616037 INFO  (searcherExecutor-1934-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 616038 INFO  (searcherExecutor-1934-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@62bf249[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 616041 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 616044 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeMixedAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSoftAndHardCommitMaxTimeMixedAdds
   [junit4]   2> 621053 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeMixedAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[529 (1559264775644381184)]} 0 3
   [junit4]   2> 621563 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 621575 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@a91e25f[collection1] main]
   [junit4]   2> 621576 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 621577 INFO  (searcherExecutor-1934-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@a91e25f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1)))}
   [junit4]   2> 622263 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 622263 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@636e9ce2
   [junit4]   2> 622264 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@24780d90[collection1] realtime]
   [junit4]   2> 622264 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 622265 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeMixedAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[530 (1559264776918401024)]} 0 1
   [junit4]   2> 622775 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 622785 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@51085bca[collection1] main]
   [junit4]   2> 622786 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 622787 INFO  (searcherExecutor-1934-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@51085bca[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1) Uninverting(_1(7.0.0):c1)))}
   [junit4]   2> 623475 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 623475 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@636e9ce2
   [junit4]   2> 623476 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5d078daf[collection1] realtime]
   [junit4]   2> 623476 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 624486 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeMixedAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSoftAndHardCommitMaxTimeMixedAdds
   [junit4]   2> 624488 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete-seed#[3332E6EB014EF6B0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSoftAndHardCommitMaxTimeDelete
   [junit4]   2> 629493 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[529 (1559264784496459776)]} 0 1
   [junit4]   2> 629493 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 629493 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@636e9ce2
   [junit4]   2> 629528 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete-seed#[3332E6EB014EF6B0]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6a9dd633[collection1] main]
   [junit4]   2> 629529 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 629530 INFO  (searcherExecutor-1934-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6a9dd633[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.0.0):c1) Uninverting(_2(7.0.0):c1)))}
   [junit4]   2> 629530 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 37
   [junit4]   2> 629531 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{delete=[529 (-1559264784537354240)]} 0 0
   [junit4]   2> 630040 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 630041 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2cfcd53b[collection1] main]
   [junit4]   2> 630041 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 630042 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[550 (1559264785073176576)]} 0 0
   [junit4]   2> 630042 INFO  (searcherExecutor-1934-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2cfcd53b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.0.0):c1)))}
   [junit4]   2> 630544 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 630555 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@67200a16[collection1] main]
   [junit4]   2> 630555 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 630556 INFO  (searcherExecutor-1934-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@67200a16[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.0.0):c1) Uninverting(_3(7.0.0):c1)))}
   [junit4]   2> 630735 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 630735 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@636e9ce2
   [junit4]   2> 630736 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@12bcab9d[collection1] realtime]
   [junit4]   2> 630736 INFO  (commitScheduler-1937-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 636746 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeDelete-seed#[3332E6EB014EF6B0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSoftAndHardCommitMaxTimeDelete
   [junit4]   2> 636749 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSoftAndHardCommitMaxTimeRapidAdds
   [junit4]   2> 641763 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[5000 (1559264797353050112)]} 0 10
   [junit4]   2> 641763 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[5001 (1559264797364584448)]} 0 0
   [junit4]   2> 641763 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[5002 (1559264797364584449)]} 0 0
   [junit4]   2> 641764 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[5003 (1559264797365633024)]} 0 0
   [junit4]   2> 641764 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[5004 (1559264797365633025)]} 0 0
   [junit4]   2> 642263 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 642766 INFO  (TEST-SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds-seed#[3332E6EB014EF6B0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSoftAndHardCommitMaxTimeRapidAdds
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SoftAutoCommitTest -Dtests.method=testSoftAndHardCommitMaxTimeRapidAdds -Dtests.seed=3332E6EB014EF6B0 -Dtests.slow=true -Dtests.locale=sq -Dtests.timezone=Antarctica/Mawson -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 6.02s J0 | SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: 1: soft wasn't fast enough
   [junit4]    > at __randomizedtesting.SeedInfo.seed([3332E6EB014EF6B0:6F2748D2EACCB7C8]:0)
   [junit4]    > at org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeRapidAdds(SoftAutoCommitTest.java:358)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 642769 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 642769 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1341231130
   [junit4]   2> 642769 INFO  (coreCloseExecutor-1939-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2979e5d5
   [junit4]   2> 642770 INFO  (coreCloseExecutor-1939-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
   [junit4]   2> 642914 INFO  (commitScheduler-1938-thread-1) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@4c0a2cbf[collection1] main]
   [junit4]   2> 642914 INFO  (coreCloseExecutor-1939-thread-1) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@636e9ce2
   [junit4]   2> 642914 ERROR (commitScheduler-1938-thread-1) [    ] o.a.s.u.CommitTracker auto commit error...:org.apache.solr.common.SolrException: Error opening new searcher
   [junit4]   2> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2002)
   [junit4]   2> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2122)
   [junit4]   2> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:636)
   [junit4]   2> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217)
   [junit4]   2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
   [junit4]   2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: openNewSearcher called on closed core
   [junit4]   2> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1989)
   [junit4]   2> ... 10 more
   [junit4]   2>
   [junit4]   2> 643700 INFO  (coreCloseExecutor-1939-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 643725 INFO  (SUITE-SoftAutoCommitTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.update.SoftAutoCommitTest_3332E6EB014EF6B0-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), subject=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), id=FST50, text=Lucene50(blocksize=128), timestamp=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Memory), intDefault=DocValuesFormat(name=Memory), range_facet_i_dv=DocValuesFormat(name=Lucene70), intDvoDefault=DocValuesFormat(name=Lucene70), range_facet_l=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=40, maxMBSortInHeap=5.293917031467684, sim=RandomSimilarity(queryNorm=true): {}, locale=sq, timezone=Antarctica/Mawson
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=186746776,total=392167424
   [junit4]   2> NOTE: All tests run in this JVM: [AnalyticsQueryTest, TestDefaultSearchFieldResource, PluginInfoTest, TestCursorMarkWithoutUniqueKey, TestStressVersions, TestSort, V2ApiIntegrationTest, TestHashPartitioner, TestDocTermOrds, TestQuerySenderNoQuery, DistributedMLTComponentTest, TestObjectReleaseTracker, JvmMetricsTest, TestCSVLoader, TestUniqueKeyFieldResource, CheckHdfsIndexTest, TestIBSimilarityFactory, SecurityConfHandlerTest, DistributedFacetPivotSmallAdvancedTest, DocumentAnalysisRequestHandlerTest, TestSolrCloudWithKerberosAlt, SolrCLIZkUtilsTest, DistanceUnitsTest, QueryElevationComponentTest, MoreLikeThisHandlerTest, AlternateDirectoryTest, AsyncCallRequestStatusResponseTest, MissingSegmentRecoveryTest, TestTolerantUpdateProcessorRandomCloud, TestFieldResource, BlockDirectoryTest, TestSolrCloudSnapshots, SolrRequestParserTest, CoreAdminRequestStatusTest, TestLockTree, MergeStrategyTest, TestCollectionAPI, PrimitiveFieldTypeTest, HdfsCollectionsAPIDistributedZkTest, DateFieldTest, CdcrReplicationHandlerTest, TestUpdate, TestPushWriter, CopyFieldTest, TestJsonFacetRefinement, TestSchemaVersionResource, TestFaceting, TestDocBasedVersionConstraints, PingRequestHandlerTest, HLLSerializationTest, TestSolrCLIRunExample, SolrGraphiteReporterTest, SyncSliceTest, PolyFieldTest, TestSlowCompositeReaderWrapper, TestConfigReload, SolrIndexConfigTest, AnalysisAfterCoreReloadTest, RequiredFieldsTest, DisMaxRequestHandlerTest, CollectionsAPISolrJTest, DOMUtilTest, TestPhraseSuggestions, TestReplicationHandlerBackup, RecoveryAfterSoftCommitTest, TestPostingsSolrHighlighter, HdfsThreadLeakTest, BlockJoinFacetSimpleTest, TestPseudoReturnFields, TestCloudDeleteByQuery, PrimUtilsTest, DeleteNodeTest, TestNRTOpen, TestRTGBase, DirectSolrConnectionTest, ChaosMonkeyNothingIsSafeTest, CursorPagingTest, TestCoreAdminApis, TestSolrJ, DocValuesMultiTest, SoftAutoCommitTest]
   [junit4] Completed [146/694 (1!)] on J0 in 28.17s, 3 tests, 1 failure <<< FAILURES!

[...truncated 1088 lines...]
   [junit4] Suite: org.apache.solr.cloud.CustomCollectionTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/init-core-data-001
   [junit4]   2> 2178473 INFO  (SUITE-CustomCollectionTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 2178480 INFO  (SUITE-CustomCollectionTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2178497 INFO  (SUITE-CustomCollectionTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001
   [junit4]   2> 2178497 INFO  (SUITE-CustomCollectionTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2178498 INFO  (Thread-3189) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2178498 INFO  (Thread-3189) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2178602 INFO  (SUITE-CustomCollectionTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:60394
   [junit4]   2> 2178652 INFO  (jetty-launcher-2232-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2178652 INFO  (jetty-launcher-2232-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2178652 INFO  (jetty-launcher-2232-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2178653 INFO  (jetty-launcher-2232-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2178658 INFO  (jetty-launcher-2232-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7f2dfbb5{/solr,null,AVAILABLE}
   [junit4]   2> 2178659 INFO  (jetty-launcher-2232-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@a47e095{SSL,[ssl, http/1.1]}{127.0.0.1:60396}
   [junit4]   2> 2178659 INFO  (jetty-launcher-2232-thread-3) [    ] o.e.j.s.Server Started @2183569ms
   [junit4]   2> 2178660 INFO  (jetty-launcher-2232-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60396}
   [junit4]   2> 2178660 ERROR (jetty-launcher-2232-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2178660 INFO  (jetty-launcher-2232-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2178660 INFO  (jetty-launcher-2232-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2178660 INFO  (jetty-launcher-2232-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2178660 INFO  (jetty-launcher-2232-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-14T00:32:06.805Z
   [junit4]   2> 2178661 INFO  (jetty-launcher-2232-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3ba514e{/solr,null,AVAILABLE}
   [junit4]   2> 2178662 INFO  (jetty-launcher-2232-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6b2d2de3{/solr,null,AVAILABLE}
   [junit4]   2> 2178662 INFO  (jetty-launcher-2232-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6226c68f{SSL,[ssl, http/1.1]}{127.0.0.1:60397}
   [junit4]   2> 2178662 INFO  (jetty-launcher-2232-thread-1) [    ] o.e.j.s.Server Started @2183572ms
   [junit4]   2> 2178662 INFO  (jetty-launcher-2232-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60397}
   [junit4]   2> 2178663 INFO  (jetty-launcher-2232-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d7cabbe{SSL,[ssl, http/1.1]}{127.0.0.1:60399}
   [junit4]   2> 2178663 ERROR (jetty-launcher-2232-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2178663 INFO  (jetty-launcher-2232-thread-4) [    ] o.e.j.s.Server Started @2183572ms
   [junit4]   2> 2178663 INFO  (jetty-launcher-2232-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60399}
   [junit4]   2> 2178663 ERROR (jetty-launcher-2232-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2178663 INFO  (jetty-launcher-2232-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2178663 INFO  (jetty-launcher-2232-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2178664 INFO  (jetty-launcher-2232-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2178664 INFO  (jetty-launcher-2232-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2178664 INFO  (jetty-launcher-2232-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2178664 INFO  (jetty-launcher-2232-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-14T00:32:06.809Z
   [junit4]   2> 2178664 INFO  (jetty-launcher-2232-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@243fff1{/solr,null,AVAILABLE}
   [junit4]   2> 2178664 INFO  (jetty-launcher-2232-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2178664 INFO  (jetty-launcher-2232-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-14T00:32:06.809Z
   [junit4]   2> 2178665 INFO  (jetty-launcher-2232-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4864b709{SSL,[ssl, http/1.1]}{127.0.0.1:60400}
   [junit4]   2> 2178665 INFO  (jetty-launcher-2232-thread-2) [    ] o.e.j.s.Server Started @2183575ms
   [junit4]   2> 2178665 INFO  (jetty-launcher-2232-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60400}
   [junit4]   2> 2178666 ERROR (jetty-launcher-2232-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2178666 INFO  (jetty-launcher-2232-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2178666 INFO  (jetty-launcher-2232-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2178666 INFO  (jetty-launcher-2232-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2178666 INFO  (jetty-launcher-2232-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2178666 INFO  (jetty-launcher-2232-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-14T00:32:06.811Z
   [junit4]   2> 2178670 INFO  (jetty-launcher-2232-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2178671 INFO  (jetty-launcher-2232-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2178672 INFO  (jetty-launcher-2232-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2178685 INFO  (jetty-launcher-2232-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60394/solr
   [junit4]   2> 2178692 INFO  (jetty-launcher-2232-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60394/solr
   [junit4]   2> 2178694 INFO  (jetty-launcher-2232-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60394/solr
   [junit4]   2> 2178699 INFO  (jetty-launcher-2232-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60394/solr
   [junit4]   2> 2178760 INFO  (jetty-launcher-2232-thread-4) [n:127.0.0.1:60399_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2178762 INFO  (jetty-launcher-2232-thread-4) [n:127.0.0.1:60399_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60399_solr
   [junit4]   2> 2178762 INFO  (jetty-launcher-2232-thread-1) [n:127.0.0.1:60397_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2178762 INFO  (jetty-launcher-2232-thread-2) [n:127.0.0.1:60400_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2178765 INFO  (jetty-launcher-2232-thread-4) [n:127.0.0.1:60399_solr    ] o.a.s.c.Overseer Overseer (id=97454150559137801-127.0.0.1:60399_solr-n_0000000000) starting
   [junit4]   2> 2178765 INFO  (jetty-launcher-2232-thread-1) [n:127.0.0.1:60397_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60397_solr
   [junit4]   2> 2178766 INFO  (jetty-launcher-2232-thread-2) [n:127.0.0.1:60400_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60400_solr
   [junit4]   2> 2178773 INFO  (zkCallback-2251-thread-1-processing-n:127.0.0.1:60399_solr) [n:127.0.0.1:60399_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2178774 INFO  (zkCallback-2249-thread-1-processing-n:127.0.0.1:60396_solr) [n:127.0.0.1:60396_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2178776 INFO  (zkCallback-2252-thread-1-processing-n:127.0.0.1:60397_solr) [n:127.0.0.1:60397_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2178776 INFO  (zkCallback-2253-thread-1-processing-n:127.0.0.1:60400_solr) [n:127.0.0.1:60400_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2178781 INFO  (jetty-launcher-2232-thread-3) [n:127.0.0.1:60396_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2178794 INFO  (jetty-launcher-2232-thread-3) [n:127.0.0.1:60396_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60396_solr
   [junit4]   2> 2178795 INFO  (jetty-launcher-2232-thread-4) [n:127.0.0.1:60399_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60399_solr
   [junit4]   2> 2178799 INFO  (zkCallback-2249-thread-1-processing-n:127.0.0.1:60396_solr) [n:127.0.0.1:60396_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2178799 INFO  (zkCallback-2253-thread-1-processing-n:127.0.0.1:60400_solr) [n:127.0.0.1:60400_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2178799 INFO  (zkCallback-2252-thread-1-processing-n:127.0.0.1:60397_solr) [n:127.0.0.1:60397_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2178800 INFO  (zkCallback-2251-thread-1-processing-n:127.0.0.1:60399_solr) [n:127.0.0.1:60399_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2178831 INFO  (zkCallback-2253-thread-1-processing-n:127.0.0.1:60400_solr) [n:127.0.0.1:60400_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2178831 INFO  (zkCallback-2251-thread-1-processing-n:127.0.0.1:60399_solr) [n:127.0.0.1:60399_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2178831 INFO  (zkCallback-2249-thread-1-processing-n:127.0.0.1:60396_solr) [n:127.0.0.1:60396_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2178833 INFO  (zkCallback-2252-thread-1-processing-n:127.0.0.1:60397_solr) [n:127.0.0.1:60397_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2178983 INFO  (jetty-launcher-2232-thread-3) [n:127.0.0.1:60396_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001/node3/.
   [junit4]   2> 2178988 INFO  (jetty-launcher-2232-thread-2) [n:127.0.0.1:60400_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001/node2/.
   [junit4]   2> 2178988 INFO  (jetty-launcher-2232-thread-4) [n:127.0.0.1:60399_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001/node4/.
   [junit4]   2> 2179088 INFO  (jetty-launcher-2232-thread-1) [n:127.0.0.1:60397_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001/node1/.
   [junit4]   2> 2183355 INFO  (SUITE-CustomCollectionTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2183358 INFO  (SUITE-CustomCollectionTest-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60394/solr ready
   [junit4]   2> 2183420 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[3332E6EB014EF6B0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateShardRepFactor
   [junit4]   2> 2183422 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[3332E6EB014EF6B0]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2183458 INFO  (qtp702008387-14854) [n:127.0.0.1:60397_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=a,b&replicationFactor=1&collection.configName=conf&name=testCreateShardRepFactor&router.name=implicit&action=CREATE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2183464 INFO  (OverseerThreadFactory-7475-thread-1-processing-n:127.0.0.1:60399_solr) [n:127.0.0.1:60399_solr    ] o.a.s.c.CreateCollectionCmd Create collection testCreateShardRepFactor
   [junit4]   2> 2183596 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_b_replica1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=b&wt=javabin&version=2
   [junit4]   2> 2183622 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_a_replica1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=a&wt=javabin&version=2
   [junit4]   2> 2183742 INFO  (zkCallback-2249-thread-1-processing-n:127.0.0.1:60396_solr) [n:127.0.0.1:60396_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2183742 INFO  (zkCallback-2252-thread-1-processing-n:127.0.0.1:60397_solr) [n:127.0.0.1:60397_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2184624 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2184645 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_b_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2184650 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2184662 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_a_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2184679 WARN  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_b_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2184681 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2184685 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_b_replica1' using configuration from collection testCreateShardRepFactor
   [junit4]   2> 2184685 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SolrCore [[testCreateShardRepFactor_b_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001/node1/testCreateShardRepFactor_b_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001/node1/./testCreateShardRepFactor_b_replica1/data/]
   [junit4]   2> 2184687 WARN  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_a_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2184688 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2184689 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_a_replica1' using configuration from collection testCreateShardRepFactor
   [junit4]   2> 2184690 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SolrCore [[testCreateShardRepFactor_a_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001/node3/testCreateShardRepFactor_a_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001/node3/./testCreateShardRepFactor_a_replica1/data/]
   [junit4]   2> 2184814 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2184815 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2184815 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2184815 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2184818 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2184818 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2184818 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2184818 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2184820 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@55ba6e11[testCreateShardRepFactor_a_replica1] main]
   [junit4]   2> 2184820 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@58634445[testCreateShardRepFactor_b_replica1] main]
   [junit4]   2> 2184823 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2184823 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2184824 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2184824 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2184824 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2184824 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2184824 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559266415575302144
   [junit4]   2> 2184824 INFO  (searcherExecutor-7485-thread-1-processing-n:127.0.0.1:60396_solr x:testCreateShardRepFactor_a_replica1 s:a c:testCreateShardRepFactor) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SolrCore [testCreateShardRepFactor_a_replica1] Registered new searcher Searcher@55ba6e11[testCreateShardRepFactor_a_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2184824 INFO  (searcherExecutor-7484-thread-1-processing-n:127.0.0.1:60397_solr x:testCreateShardRepFactor_b_replica1 s:b c:testCreateShardRepFactor) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SolrCore [testCreateShardRepFactor_b_replica1] Registered new searcher Searcher@58634445[testCreateShardRepFactor_b_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2184825 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559266415576350720
   [junit4]   2> 2184841 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2184841 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2184841 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:60396/solr/testCreateShardRepFactor_a_replica1/
   [junit4]   2> 2184841 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2184841 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SyncStrategy https://127.0.0.1:60396/solr/testCreateShardRepFactor_a_replica1/ has no replicas
   [junit4]   2> 2184841 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2184842 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2184842 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:60397/solr/testCreateShardRepFactor_b_replica1/
   [junit4]   2> 2184842 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2184842 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SyncStrategy https://127.0.0.1:60397/solr/testCreateShardRepFactor_b_replica1/ has no replicas
   [junit4]   2> 2184852 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:60396/solr/testCreateShardRepFactor_a_replica1/ a
   [junit4]   2> 2184874 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:60397/solr/testCreateShardRepFactor_b_replica1/ b
   [junit4]   2> 2184988 INFO  (zkCallback-2249-thread-1-processing-n:127.0.0.1:60396_solr) [n:127.0.0.1:60396_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2184988 INFO  (zkCallback-2252-thread-1-processing-n:127.0.0.1:60397_solr) [n:127.0.0.1:60397_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2184992 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2184999 INFO  (qtp702008387-14857) [n:127.0.0.1:60397_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_b_replica1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=b&wt=javabin&version=2} status=0 QTime=1402
   [junit4]   2> 2185049 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2185054 INFO  (qtp1702076181-14841) [n:127.0.0.1:60396_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_a_replica1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=a&wt=javabin&version=2} status=0 QTime=1432
   [junit4]   2> 2185059 INFO  (qtp702008387-14854) [n:127.0.0.1:60397_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2185161 INFO  (zkCallback-2252-thread-1-processing-n:127.0.0.1:60397_solr) [n:127.0.0.1:60397_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2185161 INFO  (zkCallback-2249-thread-1-processing-n:127.0.0.1:60396_solr) [n:127.0.0.1:60396_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2185479 INFO  (OverseerCollectionConfigSetProcessor-97454150559137801-127.0.0.1:60399_solr-n_0000000000) [n:127.0.0.1:60399_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2186068 INFO  (qtp702008387-14854) [n:127.0.0.1:60397_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={shards=a,b&replicationFactor=1&collection.configName=conf&name=testCreateShardRepFactor&router.name=implicit&action=CREATE&wt=javabin&version=2} status=0 QTime=2610
   [junit4]   2> 2186070 INFO  (qtp702008387-14846) [n:127.0.0.1:60397_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :createshard with params action=CREATESHARD&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2186074 INFO  (OverseerThreadFactory-7475-thread-2-processing-n:127.0.0.1:60399_solr) [n:127.0.0.1:60399_solr    ] o.a.s.c.CreateShardCmd Create shard invoked: {
   [junit4]   2>   "collection":"testCreateShardRepFactor",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "operation":"createshard"}
   [junit4]   2> 2186198 INFO  (zkCallback-2249-thread-1-processing-n:127.0.0.1:60396_solr) [n:127.0.0.1:60396_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2186198 INFO  (zkCallback-2252-thread-1-processing-n:127.0.0.1:60397_solr) [n:127.0.0.1:60397_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2186285 INFO  (OverseerThreadFactory-7475-thread-2-processing-n:127.0.0.1:60399_solr) [n:127.0.0.1:60399_solr    ] o.a.s.c.CreateShardCmd Creating shard testCreateShardRepFactor_x_replica1 as part of slice x of collection testCreateShardRepFactor on 127.0.0.1:60399_solr
   [junit4]   2> 2186321 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&name=testCreateShardRepFactor_x_replica1&action=CREATE&numShards=1&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2
   [junit4]   2> 2186436 INFO  (zkCallback-2249-thread-1-processing-n:127.0.0.1:60396_solr) [n:127.0.0.1:60396_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2186436 INFO  (zkCallback-2251-thread-2-processing-n:127.0.0.1:60399_solr) [n:127.0.0.1:60399_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2186436 INFO  (zkCallback-2252-thread-1-processing-n:127.0.0.1:60397_solr) [n:127.0.0.1:60397_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2187353 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2187364 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_x_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2187398 WARN  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_x_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2187399 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2187402 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_x_replica1' using configuration from collection testCreateShardRepFactor
   [junit4]   2> 2187403 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SolrCore [[testCreateShardRepFactor_x_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001/node4/testCreateShardRepFactor_x_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_3332E6EB014EF6B0-001/tempDir-001/node4/./testCreateShardRepFactor_x_replica1/data/]
   [junit4]   2> 2187512 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2187512 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2187515 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2187515 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2187517 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4f728bca[testCreateShardRepFactor_x_replica1] main]
   [junit4]   2> 2187519 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2187520 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2187520 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2187520 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559266418402263040
   [junit4]   2> 2187520 INFO  (searcherExecutor-7494-thread-1-processing-n:127.0.0.1:60399_solr x:testCreateShardRepFactor_x_replica1 s:x c:testCreateShardRepFactor) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SolrCore [testCreateShardRepFactor_x_replica1] Registered new searcher Searcher@4f728bca[testCreateShardRepFactor_x_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2187535 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2187535 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2187535 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:60399/solr/testCreateShardRepFactor_x_replica1/
   [junit4]   2> 2187535 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2187535 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SyncStrategy https://127.0.0.1:60399/solr/testCreateShardRepFactor_x_replica1/ has no replicas
   [junit4]   2> 2187546 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:60399/solr/testCreateShardRepFactor_x_replica1/ x
   [junit4]   2> 2187655 INFO  (zkCallback-2251-thread-2-processing-n:127.0.0.1:60399_solr) [n:127.0.0.1:60399_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2187656 INFO  (zkCallback-2252-thread-1-processing-n:127.0.0.1:60397_solr) [n:127.0.0.1:60397_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2187655 INFO  (zkCallback-2249-thread-1-processing-n:127.0.0.1:60396_solr) [n:127.0.0.1:60396_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2187669 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2187676 INFO  (qtp851446157-14840) [n:127.0.0.1:60399_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&name=testCreateShardRepFactor_x_replica1&action=CREATE&numShards=1&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2} status=0 QTime=1355
   [junit4]   2> 2187677 INFO  (OverseerThreadFactory-7475-thread-2-processing-n:127.0.0.1:60399_solr) [n:127.0.0.1:60399_solr    ] o.a.s.c.CreateShardCmd Finished create command on all shards for collection: testCreateShardRepFactor
   [junit4]   2> 2187681 INFO  (qtp702008387-14846) [n:127.0.0.1:60397_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CREATESHARD&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2} status=0 QTime=1611
   [junit4]   2> 2187683 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[3332E6EB014EF6B0]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateShardRepFactor
   [junit4]   2> 2187720 INFO  (TEST-CustomCollectionTest.testCustomCollectionsAPI-seed#[3332E6EB014EF6B0]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCustomCollectionsAPI
   [junit4]   2> 2187723 INFO  (TEST-CustomCollectionTest.testCustomCollectionsAPI-seed#[3332E6EB014EF6B0]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2187726 INFO  (qtp702008387-14853) [n:127.0.0.1:60397_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=testCreateShardRepFactor&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2187733 INFO  (OverseerThreadFactory-7475-thread-3-processing-n:127.0.0.1:60399_solr) [n:127.0.0.1:60399_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 2187733 INFO  (OverseerCollectionConfigSetProcessor-97454150559137801-127.0.0.1:60399_solr-n_0000000000) [n:127.0.0.1:60399_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2187735 INFO  (qtp702008387-14855) [n:127.0.0.1:60397_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.testCreateShardRepFactor.b.replica1
   [junit4]   2> 2187736 INFO  (qtp702008387-14855) [n:127.0.0.1:60397_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_b_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7f697a92
   [junit4]   2> 2187736 INFO  (qtp1702076181-14843) [n:127.0.0.1:60396_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.testCreateShardRepFactor.a.replica1
   [junit4]   2> 2187736 INFO  (qtp1702076181-14843) [n:127.0.0.1:60396_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_a_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@cb7eed6
   [junit4]   2> 2187738 INFO  (qtp702

[...truncated too long message...]

ic reporters for: solr.node
   [junit4]   2> 2987326 INFO  (TEST-TestReplicaProperties.test-seed#[3332E6EB014EF6B0]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1a295434{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2987326 INFO  (TEST-TestReplicaProperties.test-seed#[3332E6EB014EF6B0]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5c7afd1e{/,null,UNAVAILABLE}
   [junit4]   2> 2987328 INFO  (TEST-TestReplicaProperties.test-seed#[3332E6EB014EF6B0]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:62132 62132
   [junit4]   2> 2987336 INFO  (Thread-3997) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:62132 62132
   [junit4]   2> 2992473 WARN  (Thread-3997) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 7 /solr/aliases.json
   [junit4]   2> 7 /solr/clusterprops.json
   [junit4]   2> 5 /solr/security.json
   [junit4]   2> 5 /solr/configs/conf1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 7 /solr/clusterstate.json
   [junit4]   2> 5 /solr/collections/testcollection/state.json
   [junit4]   2> 4 /solr/collections/collection1/state.json
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 62 /solr/overseer/collection-queue-work
   [junit4]   2> 60 /solr/overseer/queue
   [junit4]   2> 14 /solr/overseer/queue-work
   [junit4]   2> 7 /solr/live_nodes
   [junit4]   2> 7 /solr/collections
   [junit4]   2>
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestReplicaProperties -Dtests.method=test -Dtests.seed=3332E6EB014EF6B0 -Dtests.slow=true -Dtests.locale=es -Dtests.timezone=Pacific/Samoa -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   72.5s J1 | TestReplicaProperties.test <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: timeout waiting for collection in cluster state: collection=testcollection
   [junit4]    > at __randomizedtesting.SeedInfo.seed([3332E6EB014EF6B0:BB66D931AFB29B48]:0)
   [junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForCollection(AbstractFullDistribZkTestBase.java:346)
   [junit4]    > at org.apache.solr.cloud.TestReplicaProperties.test(TestReplicaProperties.java:63)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2992476 INFO  (SUITE-TestReplicaProperties-seed#[3332E6EB014EF6B0]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestReplicaProperties_3332E6EB014EF6B0-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=888, maxMBSortInHeap=6.917873941943885, sim=RandomSimilarity(queryNorm=true): {}, locale=es, timezone=Pacific/Samoa
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=320901336,total=529530880
   [junit4]   2> NOTE: All tests run in this JVM: [ShowFileRequestHandlerTest, TestJsonFacets, CSVRequestHandlerTest, CacheHeaderTest, DocExpirationUpdateProcessorFactoryTest, WordBreakSolrSpellCheckerTest, LeaderElectionIntegrationTest, TestStandardQParsers, TestRandomFlRTGCloud, SignatureUpdateProcessorFactoryTest, TestJoin, BlockJoinFacetRandomTest, HdfsTlogReplayBufferedWhileIndexingTest, ConnectionManagerTest, UpdateParamsTest, EchoParamsTest, BaseCdcrDistributedZkTest, LeaderInitiatedRecoveryOnCommitTest, TestSolrXml, SuggestComponentContextFilterQueryTest, TestQueryUtils, DistributedFacetPivotLongTailTest, SubstringBytesRefFilterTest, TestManagedStopFilterFactory, TestSolrQueryParserResource, TestCloudInspectUtil, TestZkChroot, TestUseDocValuesAsStored, TestNonDefinedSimilarityFactory, BasicZkTest, HdfsRecoveryZkTest, SuggesterTest, TestFiltering, LeaderFailoverAfterPartitionTest, TestTrie, HdfsLockFactoryTest, TestHdfsCloudBackupRestore, TestJettySolrRunner, TestSolrIndexConfig, OverseerModifyCollectionTest, CurrencyFieldOpenExchangeTest, TestSubQueryTransformer, TestRebalanceLeaders, BasicDistributedZk2Test, OverseerStatusTest, TestPHPSerializedResponseWriter, ActionThrottleTest, TestManagedSchemaThreadSafety, SolrMetricReporterTest, TestTrieFacet, TestSchemaManager, SimpleCollectionCreateDeleteTest, LukeRequestHandlerTest, TestCodecSupport, TestRTimerTree, TestInPlaceUpdatesDistrib, TlogReplayBufferedWhileIndexingTest, HdfsWriteToMultipleCollectionsTest, SpellPossibilityIteratorTest, URLClassifyProcessorTest, TestFoldingMultitermQuery, ChaosMonkeySafeLeaderTest, TestSSLRandomization, TestHdfsBackupRestoreCore, SuggesterWFSTTest, TestFuzzyAnalyzedSuggestions, SolrCoreMetricManagerTest, TestReloadAndDeleteDocs, TestConfigSetImmutable, CoreMergeIndexesAdminHandlerTest, TestDeleteCollectionOnDownNodes, DistribDocExpirationUpdateProcessorTest, CursorMarkTest, BigEndianAscendingWordDeserializerTest, DistributedQueueTest, RAMDirectoryFactoryTest, HLLUtilTest, TestRandomFaceting, SolrCoreCheckLockOnStartupTest, JsonValidatorTest, UpdateRequestProcessorFactoryTest, TestLuceneMatchVersion, TestCustomDocTransformer, PKIAuthenticationIntegrationTest, SortByFunctionTest, RuleEngineTest, MigrateRouteKeyTest, DataDrivenBlockJoinTest, BasicAuthIntegrationTest, TestBackupRepositoryFactory, TestRealTimeGet, IndexSchemaTest, TestJmxMonitoredMap, CircularListTest, BlobRepositoryCloudTest, TestMaxScoreQueryParser, SolrInfoMBeanTest, TestPathTrie, ConvertedLegacyTest, CloudMLTQParserTest, FieldMutatingUpdateProcessorTest, HdfsDirectoryFactoryTest, SaslZkACLProviderTest, BadCopyFieldTest, TestSolrQueryResponse, SpellCheckCollatorTest, TestBulkSchemaAPI, TestRequestForwarding, ClassificationUpdateProcessorFactoryTest, TestIndexingPerformance, SmileWriterTest, TestDocSet, TestComplexPhraseQParserPlugin, ClassificationUpdateProcessorTest, DistributedIntervalFacetingTest, DeleteReplicaTest, SpellCheckComponentTest, TestInitQParser, TestQueryTypes, TestAtomicUpdateErrorCases, ChangedSchemaMergeTest, TestImpersonationWithHadoopAuth, NoCacheHeaderTest, TestSystemCollAutoCreate, CollectionsAPIAsyncDistributedZkTest, TestValueSourceCache, TestFieldCacheSortRandom, TestCollapseQParserPlugin, TestSolrCoreProperties, CachingDirectoryFactoryTest, SegmentsInfoRequestHandlerTest, TestCloudPseudoReturnFields, HighlighterConfigTest, NotRequiredUniqueKeyTest, TestManagedResourceStorage, AnalyticsMergeStrategyTest, RankQueryTest, JavabinLoaderTest, ConcurrentDeleteAndCreateCollectionTest, CoreAdminHandlerTest, HdfsBasicDistributedZkTest, TestApiFramework, TestCopyFieldCollectionResource, TestSQLHandler, TestAnalyzedSuggestions, TestDownShardTolerantSearch, TestRecovery, TestRawResponseWriter, TestSha256AuthenticationProvider, TestSearcherReuse, HttpPartitionTest, SolrGangliaReporterTest, BJQParserTest, TestComponentsName, BufferStoreTest, CdcrReplicationDistributedZkTest, TestWriterPerf, JsonLoaderTest, TestExactStatsCache, FieldAnalysisRequestHandlerTest, TestUtils, UpdateLogTest, TestExceedMaxTermLength, ResponseLogComponentTest, ExternalFileFieldSortTest, InfixSuggestersTest, TestSolrQueryParser, TestLMDirichletSimilarityFactory, TestDistribIDF, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, LeaderElectionTest, ZkSolrClientTest, ShardRoutingCustomTest, AutoCommitTest, TestDistributedGrouping, DistributedSpellCheckComponentTest, TestStressReorder, TestReload, DistributedTermsComponentTest, SolrCoreTest, StatsComponentTest, BadIndexSchemaTest, TestFunctionQuery, BasicFunctionalityTest, DirectUpdateHandlerTest, TestBadConfig, TestIndexSearcher, DistributedQueryElevationComponentTest, DocValuesTest, DebugComponentTest, IndexBasedSpellCheckerTest, DirectSolrSpellCheckerTest, TestOmitPositions, FileBasedSpellCheckerTest, RequestHandlersTest, FastVectorHighlighterTest, LoggingHandlerTest, TestJmxIntegration, BinaryUpdateRequestHandlerTest, SOLR749Test, BadComponentTest, TestMergePolicyConfig, TestSolrDeletionPolicy2, MultiTermTest, SampleTest, TestBinaryField, TestSearchPerf, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestCharFilters, SynonymTokenizerTest, TestXIncludeConfig, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, ResourceLoaderTest, TestFastOutputStream, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, TestSuggestSpellingConverter, SpellingQueryConverterTest, ClusterStateTest, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, UUIDFieldTest, TestCrossCoreJoin, TestDistributedMissingSort, TestEmbeddedSolrServerConstructors, ConnectionReuseTest, CdcrRequestHandlerTest, CdcrVersionReplicationTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionStateFormat2Test, CustomCollectionTest, DocValuesNotIndexedTest, LeaderElectionContextKeyTest, MultiThreadedOCPTest, OverseerCollectionConfigSetProcessorTest, OverseerRolesTest, PeerSyncReplicationTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, SolrCloudExampleTest, TestAuthenticationFramework, TestCloudPivotFacet, TestClusterProperties, TestConfigSetsAPI, TestConfigSetsAPIZkFailure, TestCryptoKeys, TestLeaderElectionZkExpiry, TestMiniSolrCloudCluster, TestReplicaProperties]
   [junit4] Completed [517/694 (4!)] on J1 in 72.51s, 1 test, 1 error <<< FAILURES!

[...truncated 63145 lines...]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3831 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3831/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

339 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.AnalysisAfterCoreReloadTest

Error Message:
org.apache.solr.AnalysisAfterCoreReloadTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.AnalysisAfterCoreReloadTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/AnalysisAfterCoreReloadTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.ConvertedLegacyTest

Error Message:
org.apache.solr.ConvertedLegacyTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.ConvertedLegacyTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/ConvertedLegacyTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.CursorPagingTest

Error Message:
invalid API spec: apispec/cluster.security.authorization.json

Stack Trace:
java.lang.RuntimeException: invalid API spec: apispec/cluster.security.authorization.json
        at __randomizedtesting.SeedInfo.seed([6251E2874C421FF1]:0)
        at org.apache.solr.common.util.ValidatingJsonMap.parse(ValidatingJsonMap.java:318)
        at org.apache.solr.api.ApiBag.lambda$getSpec$0(ApiBag.java:229)
        at org.apache.solr.api.Api.getSpec(Api.java:64)
        at org.apache.solr.api.ApiBag.register(ApiBag.java:72)
        at org.apache.solr.core.PluginBag.put(PluginBag.java:215)
        at org.apache.solr.core.PluginBag.put(PluginBag.java:186)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:516)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:177)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:140)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:146)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:109)
        at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:741)
        at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:731)
        at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:559)
        at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:548)
        at org.apache.solr.CursorPagingTest.beforeTests(CursorPagingTest.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.CursorPagingTest

Error Message:
ObjectTracker found 2 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:195)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:484)  at org.apache.solr.util.TestHarness.<init>(TestHarness.java:177)  at org.apache.solr.util.TestHarness.<init>(TestHarness.java:140)  at org.apache.solr.util.TestHarness.<init>(TestHarness.java:146)  at org.apache.solr.util.TestHarness.<init>(TestHarness.java:109)  at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:741)  at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:731)  at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:559)  at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:548)  at org.apache.solr.CursorPagingTest.beforeTests(CursorPagingTest.java:68)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.lang.reflect.Method.invoke(Method.java:498)  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:90)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:490)  at org.apache.solr.util.TestHarness.<init>(TestHarness.java:177)  at org.apache.solr.util.TestHarness.<init>(TestHarness.java:140)  at org.apache.solr.util.TestHarness.<init>(TestHarness.java:146)  at org.apache.solr.util.TestHarness.<init>(TestHarness.java:109)  at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:741)  at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:731)  at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:559)  at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:548)  at org.apache.solr.CursorPagingTest.beforeTests(CursorPagingTest.java:68)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.lang.reflect.Method.invoke(Method.java:498)  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)  at java.lang.Thread.run(Thread.java:745)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)
        at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:195)
        at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:484)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:177)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:140)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:146)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:109)
        at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:741)
        at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:731)
        at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:559)
        at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:548)
        at org.apache.solr.CursorPagingTest.beforeTests(CursorPagingTest.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)
        at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:90)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:490)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:177)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:140)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:146)
        at org.apache.solr.util.TestHarness.<init>(TestHarness.java:109)
        at org.apache.solr.SolrTestCaseJ4.createCore(SolrTestCaseJ4.java:741)
        at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:731)
        at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:559)
        at org.apache.solr.SolrTestCaseJ4.initCore(SolrTestCaseJ4.java:548)
        at org.apache.solr.CursorPagingTest.beforeTests(CursorPagingTest.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


        at __randomizedtesting.SeedInfo.seed([6251E2874C421FF1]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:301)
        at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.CursorPagingTest

Error Message:
Could not remove the following files (in the order of attempts):    /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.CursorPagingTest_6251E2874C421FF1-001/init-core-data-001: java.nio.file.FileSystemException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.CursorPagingTest_6251E2874C421FF1-001/init-core-data-001: Too many open files    /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.CursorPagingTest_6251E2874C421FF1-001: java.nio.file.FileSystemException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.CursorPagingTest_6251E2874C421FF1-001: Too many open files

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
   /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.CursorPagingTest_6251E2874C421FF1-001/init-core-data-001: java.nio.file.FileSystemException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.CursorPagingTest_6251E2874C421FF1-001/init-core-data-001: Too many open files
   /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.CursorPagingTest_6251E2874C421FF1-001: java.nio.file.FileSystemException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.CursorPagingTest_6251E2874C421FF1-001: Too many open files

        at org.apache.lucene.util.IOUtils.rm(IOUtils.java:323)
        at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
        at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.CursorPagingTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.CursorPagingTest:     1) Thread[id=8301, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CursorPagingTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=8300, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CursorPagingTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.CursorPagingTest:
   1) Thread[id=8301, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CursorPagingTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=8300, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CursorPagingTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([6251E2874C421FF1]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.DisMaxRequestHandlerTest

Error Message:
org.apache.solr.DisMaxRequestHandlerTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.DisMaxRequestHandlerTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/DisMaxRequestHandlerTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.EchoParamsTest

Error Message:
org.apache.solr.EchoParamsTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.EchoParamsTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/EchoParamsTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.MinimalSchemaTest

Error Message:
org.apache.solr.MinimalSchemaTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.MinimalSchemaTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/MinimalSchemaTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.SampleTest

Error Message:
org.apache.solr.SampleTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.SampleTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/SampleTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.SolrInfoMBeanTest

Error Message:
org.apache.solr.SolrInfoMBeanTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.SolrInfoMBeanTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/SolrInfoMBeanTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.TestDistributedGrouping

Error Message:
org.apache.solr.TestDistributedGrouping

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.TestDistributedGrouping
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/TestDistributedGrouping.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.TestDistributedSearch

Error Message:
org.apache.solr.TestDistributedSearch

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.TestDistributedSearch
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/TestDistributedSearch.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.TestDocumentBuilder

Error Message:
org.apache.solr.TestDocumentBuilder

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.TestDocumentBuilder
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/TestDocumentBuilder.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.TestRandomDVFaceting

Error Message:
org.apache.solr.TestRandomDVFaceting

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.TestRandomDVFaceting
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/TestRandomDVFaceting.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.TestRandomFaceting

Error Message:
org.apache.solr.TestRandomFaceting

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.TestRandomFaceting
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/TestRandomFaceting.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.analysis.TestLuceneMatchVersion

Error Message:
org.apache.solr.analysis.TestLuceneMatchVersion

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.analysis.TestLuceneMatchVersion
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/analysis/TestLuceneMatchVersion.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.analysis.TestReversedWildcardFilterFactory

Error Message:
org.apache.solr.analysis.TestReversedWildcardFilterFactory

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.analysis.TestReversedWildcardFilterFactory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/analysis/TestReversedWildcardFilterFactory.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
org.apache.solr.cloud.BasicDistributedZkTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.BasicDistributedZkTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/BasicDistributedZkTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/ChaosMonkeySafeLeaderTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ClusterStateTest

Error Message:
org.apache.solr.cloud.ClusterStateTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.ClusterStateTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/ClusterStateTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ClusterStateUpdateTest

Error Message:
org.apache.solr.cloud.ClusterStateUpdateTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.ClusterStateUpdateTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/ClusterStateUpdateTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionReloadTest

Error Message:
org.apache.solr.cloud.CollectionReloadTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.CollectionReloadTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/CollectionReloadTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionStateFormat2Test

Error Message:
org.apache.solr.cloud.CollectionStateFormat2Test

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.CollectionStateFormat2Test
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/CollectionStateFormat2Test.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest

Error Message:
org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/CollectionsAPIAsyncDistributedZkTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
Error from server at https://127.0.0.1:62285/solr/awhollynewcollection_0: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 510 </title> </head> <body> <h2>HTTP ERROR: 510</h2> <p>Problem accessing /solr/awhollynewcollection_0/select. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg={&quot;awhollynewcollection_0&quot;:6},code=510}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/> </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:62285/solr/awhollynewcollection_0: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 510 </title>
</head>
<body>
<h2>HTTP ERROR: 510</h2>
<p>Problem accessing /solr/awhollynewcollection_0/select. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg={&quot;awhollynewcollection_0&quot;:6},code=510}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/>
</body>
</html>

        at __randomizedtesting.SeedInfo.seed([6251E2874C421FF1:2A2496334A713064]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:439)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:391)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1358)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1109)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1212)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1212)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1212)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1212)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1212)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:523)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPISolrJTest

Error Message:
org.apache.solr.cloud.CollectionsAPISolrJTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.CollectionsAPISolrJTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/CollectionsAPISolrJTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/ConcurrentDeleteAndCreateCollectionTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConfigSetsAPITest

Error Message:
org.apache.solr.cloud.ConfigSetsAPITest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.ConfigSetsAPITest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/ConfigSetsAPITest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConnectionManagerTest

Error Message:
org.apache.solr.cloud.ConnectionManagerTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.ConnectionManagerTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/ConnectionManagerTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CustomCollectionTest

Error Message:
org.apache.solr.cloud.CustomCollectionTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.CustomCollectionTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/CustomCollectionTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteInactiveReplicaTest

Error Message:
org.apache.solr.cloud.DeleteInactiveReplicaTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.DeleteInactiveReplicaTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/DeleteInactiveReplicaTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteNodeTest

Error Message:
org.apache.solr.cloud.DeleteNodeTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.DeleteNodeTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/DeleteNodeTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteShardTest

Error Message:
org.apache.solr.cloud.DeleteShardTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.DeleteShardTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/DeleteShardTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DistribCursorPagingTest

Error Message:
org.apache.solr.cloud.DistribCursorPagingTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.DistribCursorPagingTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/DistribCursorPagingTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest

Error Message:
org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/DistribDocExpirationUpdateProcessorTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DistribJoinFromCollectionTest

Error Message:
org.apache.solr.cloud.DistribJoinFromCollectionTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.DistribJoinFromCollectionTest
        at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:273)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:233)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/cloud/DistribJoinFromCollectionTest.class (Too many open files)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
        at sun.misc.Resource.cachedInputStream(Resource.java:77)
        at sun.misc.Resource.getByteBuffer(Resource.java:160)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        ... 12 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Stack Trace:
java.lang.ClassNotFoundException: org.apache.solr.cloud.FullSolrCloudD

[...truncated too long message...]

Loader.java:368)
   [junit4]    > ... 12 more
   [junit4] Completed [688/694 (321!)] on J1 in 0.00s, 0 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.util.TestSolrCLIRunExample
   [junit4] ERROR   0.00s J1 | TestSolrCLIRunExample (suite) <<<
   [junit4]    > Throwable #1: java.lang.ClassNotFoundException: org.apache.solr.util.TestSolrCLIRunExample
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
   [junit4]    > at java.security.AccessController.doPrivileged(Native Method)
   [junit4]    > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
   [junit4]    > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
   [junit4]    > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
   [junit4]    > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
   [junit4]    > at java.lang.Class.forName0(Native Method)
   [junit4]    > at java.lang.Class.forName(Class.java:348)
   [junit4]    > Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/util/TestSolrCLIRunExample.class (Too many open files)
   [junit4]    > at java.io.FileInputStream.open0(Native Method)
   [junit4]    > at java.io.FileInputStream.open(FileInputStream.java:195)
   [junit4]    > at java.io.FileInputStream.<init>(FileInputStream.java:138)
   [junit4]    > at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
   [junit4]    > at sun.misc.Resource.cachedInputStream(Resource.java:77)
   [junit4]    > at sun.misc.Resource.getByteBuffer(Resource.java:160)
   [junit4]    > at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
   [junit4]    > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
   [junit4]    > ... 12 more
   [junit4] Completed [689/694 (322!)] on J1 in 0.00s, 0 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.util.UtilsToolTest
   [junit4] ERROR   0.00s J1 | UtilsToolTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.ClassNotFoundException: org.apache.solr.util.UtilsToolTest
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
   [junit4]    > at java.security.AccessController.doPrivileged(Native Method)
   [junit4]    > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
   [junit4]    > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
   [junit4]    > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
   [junit4]    > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
   [junit4]    > at java.lang.Class.forName0(Native Method)
   [junit4]    > at java.lang.Class.forName(Class.java:348)
   [junit4]    > Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/util/UtilsToolTest.class (Too many open files)
   [junit4]    > at java.io.FileInputStream.open0(Native Method)
   [junit4]    > at java.io.FileInputStream.open(FileInputStream.java:195)
   [junit4]    > at java.io.FileInputStream.<init>(FileInputStream.java:138)
   [junit4]    > at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
   [junit4]    > at sun.misc.Resource.cachedInputStream(Resource.java:77)
   [junit4]    > at sun.misc.Resource.getByteBuffer(Resource.java:160)
   [junit4]    > at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
   [junit4]    > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
   [junit4]    > ... 12 more
   [junit4] Completed [690/694 (323!)] on J1 in 0.00s, 0 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.util.hll.BigEndianAscendingWordSerializerTest
   [junit4] ERROR   0.00s J1 | BigEndianAscendingWordSerializerTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.ClassNotFoundException: org.apache.solr.util.hll.BigEndianAscendingWordSerializerTest
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
   [junit4]    > at java.security.AccessController.doPrivileged(Native Method)
   [junit4]    > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
   [junit4]    > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
   [junit4]    > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
   [junit4]    > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
   [junit4]    > at java.lang.Class.forName0(Native Method)
   [junit4]    > at java.lang.Class.forName(Class.java:348)
   [junit4]    > Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/util/hll/BigEndianAscendingWordSerializerTest.class (Too many open files)
   [junit4]    > at java.io.FileInputStream.open0(Native Method)
   [junit4]    > at java.io.FileInputStream.open(FileInputStream.java:195)
   [junit4]    > at java.io.FileInputStream.<init>(FileInputStream.java:138)
   [junit4]    > at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
   [junit4]    > at sun.misc.Resource.cachedInputStream(Resource.java:77)
   [junit4]    > at sun.misc.Resource.getByteBuffer(Resource.java:160)
   [junit4]    > at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
   [junit4]    > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
   [junit4]    > ... 12 more
   [junit4] Completed [691/694 (324!)] on J1 in 0.00s, 0 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.util.hll.ExplicitHLLTest
   [junit4] ERROR   0.00s J1 | ExplicitHLLTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.ClassNotFoundException: org.apache.solr.util.hll.ExplicitHLLTest
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
   [junit4]    > at java.security.AccessController.doPrivileged(Native Method)
   [junit4]    > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
   [junit4]    > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
   [junit4]    > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
   [junit4]    > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
   [junit4]    > at java.lang.Class.forName0(Native Method)
   [junit4]    > at java.lang.Class.forName(Class.java:348)
   [junit4]    > Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/util/hll/ExplicitHLLTest.class (Too many open files)
   [junit4]    > at java.io.FileInputStream.open0(Native Method)
   [junit4]    > at java.io.FileInputStream.open(FileInputStream.java:195)
   [junit4]    > at java.io.FileInputStream.<init>(FileInputStream.java:138)
   [junit4]    > at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
   [junit4]    > at sun.misc.Resource.cachedInputStream(Resource.java:77)
   [junit4]    > at sun.misc.Resource.getByteBuffer(Resource.java:160)
   [junit4]    > at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
   [junit4]    > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
   [junit4]    > ... 12 more
   [junit4] Completed [692/694 (325!)] on J1 in 0.00s, 0 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.util.hll.HLLSerializationTest
   [junit4] ERROR   0.00s J1 | HLLSerializationTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.ClassNotFoundException: org.apache.solr.util.hll.HLLSerializationTest
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:370)
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
   [junit4]    > at java.security.AccessController.doPrivileged(Native Method)
   [junit4]    > at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
   [junit4]    > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
   [junit4]    > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
   [junit4]    > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
   [junit4]    > at java.lang.Class.forName0(Native Method)
   [junit4]    > at java.lang.Class.forName(Class.java:348)
   [junit4]    > Caused by: java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test/org/apache/solr/util/hll/HLLSerializationTest.class (Too many open files)
   [junit4]    > at java.io.FileInputStream.open0(Native Method)
   [junit4]    > at java.io.FileInputStream.open(FileInputStream.java:195)
   [junit4]    > at java.io.FileInputStream.<init>(FileInputStream.java:138)
   [junit4]    > at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1288)
   [junit4]    > at sun.misc.Resource.cachedInputStream(Resource.java:77)
   [junit4]    > at sun.misc.Resource.getByteBuffer(Resource.java:160)
   [junit4]    > at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
   [junit4]    > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
   [junit4]    > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
   [junit4]    > ... 12 more
   [junit4] Completed [693/694 (326!)] on J1 in 0.00s, 0 tests, 1 error <<< FAILURES!

[...truncated 62597 lines...]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3832 - Failure!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3832/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

No tests ran.

Build Log:
[...truncated 595 lines...]
ERROR: command execution failed.
ERROR: Step ‘Archive the artifacts’ failed: no workspace for Lucene-Solr-master-MacOSX #3832
ERROR: Step ‘Scan for compiler warnings’ failed: no workspace for Lucene-Solr-master-MacOSX #3832
ERROR: Step ‘Publish JUnit test result report’ failed: no workspace for Lucene-Solr-master-MacOSX #3832
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3833 - Still Unstable!

Policeman Jenkins Server-2
In reply to this post by Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3833/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

11 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testCreateShardRepFactor

Error Message:
Error from server at https://127.0.0.1:54548/solr: Could not find collection : testCreateShardRepFactor

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:54548/solr: Could not find collection : testCreateShardRepFactor
        at __randomizedtesting.SeedInfo.seed([BA3F445A78D88886:4ED1CEDE608C9D46]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:439)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:391)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1358)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1109)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
        at org.apache.solr.cloud.CustomCollectionTest.testCreateShardRepFactor(CustomCollectionTest.java:190)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter

Error Message:
Collection not found: withShardField

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: withShardField
        at __randomizedtesting.SeedInfo.seed([BA3F445A78D88886:EF6FACC8D4214776]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1376)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1072)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:232)
        at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.test

Error Message:
IOException occured when talking to server at: http://127.0.0.1:54750/solr/expiry_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:54750/solr/expiry_shard1_replica1
        at __randomizedtesting.SeedInfo.seed([BA3F445A78D88886:326B7B80D624E57E]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:641)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.getIndexVersionOfAllReplicas(DistribDocExpirationUpdateProcessorTest.java:149)
        at org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.test(DistribDocExpirationUpdateProcessorTest.java:81)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:54750 failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:532)
        ... 44 more


FAILED:  org.apache.solr.cloud.DistributedVersionInfoTest.testReplicaVersionHandling

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
        at __randomizedtesting.SeedInfo.seed([BA3F445A78D88886:66C693A0DAA342C7]:0)
        at org.apache.solr.common.cloud.ZkStateReader.waitForState(ZkStateReader.java:1251)
        at org.apache.solr.cloud.DistributedVersionInfoTest.testReplicaVersionHandling(DistributedVersionInfoTest.java:86)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.common.SolrException: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at __randomizedtesting.SeedInfo.seed([BA3F445A78D88886:326B7B80D624E57E]:0)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:170)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:116)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:106)
        at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:226)
        at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:615)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:268)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:330)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:983)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1782)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
        at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:43)
        at org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
        at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:145)
        ... 40 more
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
        at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
        at java.nio.channels.Selector.open(Selector.java:227)
        at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
        at sun.reflect.GeneratedConstructorAccessor9.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1779)
        ... 46 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.LeaderElectionContextKeyTest

Error Message:
Could not find collection:testCollection2

Stack Trace:
java.lang.AssertionError: Could not find collection:testCollection2
        at __randomizedtesting.SeedInfo.seed([BA3F445A78D88886]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
        at org.apache.solr.cloud.LeaderElectionContextKeyTest.setupCluster(LeaderElectionContextKeyTest.java:58)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest.test

Error Message:


Stack Trace:
java.lang.NullPointerException
        at __randomizedtesting.SeedInfo.seed([BA3F445A78D88886:326B7B80D624E57E]:0)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1860)
        at org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest.oneShardTest(LeaderInitiatedRecoveryOnCommitTest.java:113)
        at org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest.test(LeaderInitiatedRecoveryOnCommitTest.java:49)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.OverseerRolesTest

Error Message:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.common.SolrException: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at __randomizedtesting.SeedInfo.seed([BA3F445A78D88886]:0)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:170)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:116)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:111)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:98)
        at org.apache.solr.cloud.MiniSolrCloudCluster.waitForAllNodes(MiniSolrCloudCluster.java:260)
        at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:254)
        at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:188)
        at org.apache.solr.cloud.OverseerRolesTest.setupCluster(OverseerRolesTest.java:50)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1782)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
        at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:43)
        at org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
        at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:145)
        ... 31 more
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
        at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
        at java.nio.channels.Selector.open(Selector.java:227)
        at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
        at sun.reflect.GeneratedConstructorAccessor9.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1779)
        ... 37 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.OverseerRolesTest

Error Message:
ObjectTracker found 17 object(s) that were not released!!! [ZkController, InternalHttpClient, ZkController, SolrZkClient, SolrZkClient, ZkController, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, SolrZkClient, InternalHttpClient, Overseer, InternalHttpClient, InternalHttpClient, ZkController, SolrZkClient] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:422)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:195)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:484)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:422)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:184)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:297)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:184)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:297)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:422)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:90)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:490)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:195)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:484)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:90)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:490)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:195)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:484)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:184)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:297)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:90)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:490)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.Overseer.start(Overseer.java:523)  at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:747)  at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)  at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)  at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)  at org.apache.solr.cloud.ZkController.init(ZkController.java:688)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:420)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)  at org.apache.solr.update.UpdateShardHandler.<init>(UpdateShardHandler.java:90)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:490)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)  at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:195)  at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:484)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:422)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:184)  at org.apache.solr.cloud.ZkController.<init>(ZkController.java:297)  at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)  at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)  at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)  at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)  at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)  at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)  at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)  at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)  at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)  at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)  at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)  at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)  at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 17 object(s) that were not released!!! [ZkController, InternalHttpClient, ZkController, SolrZkClient, SolrZkClient, ZkController, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, SolrZkClient, InternalHttpClient, Overseer, InternalHttpClient, InternalHttpClient, ZkController, SolrZkClient]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.cloud.ZkController.<init>(ZkController.java:422)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:287)
        at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:195)
        at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:47)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:484)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.cloud.ZkController.<init>(ZkController.java:422)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:184)
        at org.apache.solr.cloud.ZkController.<init>(ZkController.java:297)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:184)
        at org.apache.solr.cloud.ZkController.<init>(ZkController.java:297)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1565)
        at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1599)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1285)
        at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1130)
        at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
        at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:376)
        at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:239)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
        at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at org.apache.solr.cloud.ZkController.<init>(ZkController.java:422)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:499)
        at org.apache.solr.ser

[...truncated too long message...]

        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    5) Thread[id=14483, name=qtp1032941155-14483, state=RUNNABLE, group=TGRP-OverseerRolesTest]
   [junit4]    >         at sun.nio.ch.IOUtil.drain(Native Method)
   [junit4]    >         at sun.nio.ch.KQueueSelectorImpl.updateSelectedKeys(KQueueSelectorImpl.java:181)
   [junit4]    >         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    6) Thread[id=14485, name=qtp2033424983-14485, state=RUNNABLE, group=TGRP-OverseerRolesTest]
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.interrupt(Native Method)
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.interrupt(KQueueArrayWrapper.java:203)
   [junit4]    >         at sun.nio.ch.KQueueSelectorImpl.wakeup(KQueueSelectorImpl.java:257)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]    >         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:116)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    7) Thread[id=14554, name=jetty-launcher-2592-thread-1-SendThread(127.0.0.1:55036), state=TIMED_WAITING, group=TGRP-OverseerRolesTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >    8) Thread[id=14552, name=jetty-launcher-2592-thread-3-SendThread(127.0.0.1:55036), state=TIMED_WAITING, group=TGRP-OverseerRolesTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    9) Thread[id=14478, name=qtp217954388-14478, state=RUNNABLE, group=TGRP-OverseerRolesTest]
   [junit4]    >         at sun.nio.ch.IOUtil.drain(Native Method)
   [junit4]    >         at sun.nio.ch.KQueueSelectorImpl.updateSelectedKeys(KQueueSelectorImpl.java:181)
   [junit4]    >         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]    >         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
   [junit4]    >         at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   10) Thread[id=14506, name=org.eclipse.jetty.server.session.HashSessionManager@2b9c62f7Timer, state=TIMED_WAITING, group=TGRP-OverseerRolesTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   11) Thread[id=14547, name=jetty-launcher-2592-thread-4-SendThread(127.0.0.1:55036), state=TIMED_WAITING, group=TGRP-OverseerRolesTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >   12) Thread[id=14517, name=org.eclipse.jetty.server.session.HashSessionManager@3fe877bbTimer, state=TIMED_WAITING, group=TGRP-OverseerRolesTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([BA3F445A78D88886]:0)
   [junit4] Completed [503/694 (7!)] on J0 in 147.47s, 0 tests, 1 failure, 3 errors <<< FAILURES!

[...truncated 63201 lines...]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3834 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3834/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

10 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Could not find collection:collection2

Stack Trace:
java.lang.AssertionError: Could not find collection:collection2
        at __randomizedtesting.SeedInfo.seed([D08C0DB04A5A4B25:58D8326AE4A626DD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:144)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:620)
        at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:
Task 3001 did not complete, final state: FAILED expected same:<COMPLETED> was not:<FAILED>

Stack Trace:
java.lang.AssertionError: Task 3001 did not complete, final state: FAILED expected same:<COMPLETED> was not:<FAILED>
        at __randomizedtesting.SeedInfo.seed([D08C0DB04A5A4B25:58D8326AE4A626DD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotSame(Assert.java:641)
        at org.junit.Assert.assertSame(Assert.java:580)
        at org.apache.solr.cloud.MultiThreadedOCPTest.testDeduplicationOfSubmittedTasks(MultiThreadedOCPTest.java:250)
        at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.common.SolrException: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at __randomizedtesting.SeedInfo.seed([D08C0DB04A5A4B25:58D8326AE4A626DD]:0)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:170)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:116)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:106)
        at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:226)
        at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:615)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:268)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:330)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:983)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1782)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
        at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:43)
        at org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
        at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:145)
        ... 40 more
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
        at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
        at java.nio.channels.Selector.open(Selector.java:227)
        at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
        at sun.reflect.GeneratedConstructorAccessor252.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1779)
        ... 46 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ShardSplitTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest:     1) Thread[id=19862, name=TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D08C0DB04A5A4B25]-EventThread, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    2) Thread[id=19853, name=Thread-3634, state=WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1249)         at java.lang.Thread.join(Thread.java:1323)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)    3) Thread[id=19861, name=TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D08C0DB04A5A4B25]-SendThread(127.0.0.1:64006), state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)    4) Thread[id=19867, name=Thread-3637, state=WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1249)         at java.lang.Thread.join(Thread.java:1323)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)    5) Thread[id=19871, name=ProcessThread(sid:0 cport:-1):, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)    6) Thread[id=19857, name=ProcessThread(sid:0 cport:-1):, state=WAITING, group=TGRP-ShardSplitTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)    7) Thread[id=19855, name=SessionTracker, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)    8) Thread[id=19869, name=SessionTracker, state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)    9) Thread[id=19868, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=19854, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest:
   1) Thread[id=19862, name=TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D08C0DB04A5A4B25]-EventThread, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   2) Thread[id=19853, name=Thread-3634, state=WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1249)
        at java.lang.Thread.join(Thread.java:1323)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302)
        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   3) Thread[id=19861, name=TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D08C0DB04A5A4B25]-SendThread(127.0.0.1:64006), state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   4) Thread[id=19867, name=Thread-3637, state=WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1249)
        at java.lang.Thread.join(Thread.java:1323)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302)
        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   5) Thread[id=19871, name=ProcessThread(sid:0 cport:-1):, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)
   6) Thread[id=19857, name=ProcessThread(sid:0 cport:-1):, state=WAITING, group=TGRP-ShardSplitTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)
   7) Thread[id=19855, name=SessionTracker, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Object.wait(Native Method)
        at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   8) Thread[id=19869, name=SessionTracker, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Object.wait(Native Method)
        at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   9) Thread[id=19868, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=19854, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([D08C0DB04A5A4B25]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ShardSplitTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=19861, name=TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D08C0DB04A5A4B25]-SendThread(127.0.0.1:64006), state=TIMED_WAITING, group=TGRP-ShardSplitTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    2) Thread[id=19868, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=19861, name=TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D08C0DB04A5A4B25]-SendThread(127.0.0.1:64006), state=TIMED_WAITING, group=TGRP-ShardSplitTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   2) Thread[id=19868, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([D08C0DB04A5A4B25]:0)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit

Error Message:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.common.SolrException: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at __randomizedtesting.SeedInfo.seed([D08C0DB04A5A4B25:29C19E1F762F06AF]:0)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:170)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:116)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:106)
        at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:226)
        at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:121)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:615)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.initCloud(AbstractFullDistribZkTestBase.java:268)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:330)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:983)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1782)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
        at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:43)
        at org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
        at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:145)
        ... 40 more
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
        at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
        at java.nio.channels.Selector.open(Selector.java:227)
        at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
        at sun.reflect.GeneratedConstructorAccessor252.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1779)
        ... 46 more


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey

Error Message:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml (Too many open files)

Stack Trace:
java.io.FileNotFoundException: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml (Too many open files)
        at __randomizedtesting.SeedInfo.seed([D08C0DB04A5A4B25:5BABDE610B5CE0A1]:0)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:294)
        at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1815)
        at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:419)
        at org.apache.solr.cloud.AbstractZkTestCase.putConfig(AbstractZkTestCase.java:133)
        at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:102)
        at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:80)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.distribSetUp(AbstractDistribZkTestBase.java:80)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribSetUp(AbstractFullDistribZkTestBase.java:219)
        at org.apache.solr.cloud.ShardSplitTest.distribSetUp(ShardSplitTest.java:85)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:955)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitShardWithRule

Error Message:
Could not get the port for ZooKeeper server

Stack Trace:
java.lang.RuntimeException: Could not get the port for ZooKeeper server
        at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:515)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.distribSetUp(AbstractDistribZkTestBase.java:70)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribSetUp(AbstractFullDistribZkTestBase.java:219)
        at org.apache.solr.cloud.ShardSplitTest.distribSetUp(ShardSplitTest.java:85)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:955)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ShardSplitTest.test

Error Message:
Could not get the port for ZooKeeper server

Stack Trace:
java.lang.RuntimeException: Could not get the port for ZooKeeper server
        at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:515)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.distribSetUp(AbstractDistribZkTestBase.java:70)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribSetUp(AbstractFullDistribZkTestBase.java:219)
        at org.apache.solr.cloud.ShardSplitTest.distribSetUp(ShardSplitTest.java:85)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:955)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitStaticIndexReplication

Error Message:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.common.SolrException: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at __randomizedtesting.SeedInfo.seed([D08C0DB04A5A4B25:9AC699B0DBF26420]:0)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:170)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:116)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:106)
        at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:86)
        at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:80)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.distribSetUp(AbstractDistribZkTestBase.java:80)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribSetUp(AbstractFullDistribZkTestBase.java:219)
        at org.apache.solr.cloud.ShardSplitTest.distribSetUp(ShardSplitTest.java:85)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:955)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1782)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
        at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:43)
        at org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
        at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
        at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:145)
        ... 39 more
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method)
        at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
        at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
        at java.nio.channels.Selector.open(Selector.java:227)
        at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
        at sun.reflect.GeneratedConstructorAccessor252.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1779)
        ... 45 more




Build Log:
[...truncated 11910 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/init-core-data-001
   [junit4]   2> 1951834 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[D08C0DB04A5A4B25]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1951838 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[D08C0DB04A5A4B25]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1951838 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[D08C0DB04A5A4B25]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1951839 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1951840 INFO  (Thread-2892) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1951840 INFO  (Thread-2892) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1951940 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.ZkTestServer start zk server on port:62519
   [junit4]   2> 1951966 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1951971 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1951983 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1951986 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1951990 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1951993 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1951996 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1952010 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1952017 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1952026 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1952029 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1953081 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/control-001/cores/collection1
   [junit4]   2> 1953084 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1953086 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@62bef680{/,null,AVAILABLE}
   [junit4]   2> 1953088 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@449e4f24{HTTP/1.1,[http/1.1]}{127.0.0.1:62522}
   [junit4]   2> 1953088 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.Server Started @1959075ms
   [junit4]   2> 1953088 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/tempDir-001/control/data, hostContext=/, hostPort=62522, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/control-001/cores}
   [junit4]   2> 1953088 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1953089 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1953089 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1953089 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1953089 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-15T09:30:13.809Z
   [junit4]   2> 1953093 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1953093 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/control-001/solr.xml
   [junit4]   2> 1953105 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:62519/solr
   [junit4]   2> 1953151 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62522_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1953153 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62522_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:62522_
   [junit4]   2> 1953155 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62522_    ] o.a.s.c.Overseer Overseer (id=97461928759263236-127.0.0.1:62522_-n_0000000000) starting
   [junit4]   2> 1953170 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62522_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:62522_
   [junit4]   2> 1953174 INFO  (zkCallback-3091-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1953293 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62522_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/control-001/cores
   [junit4]   2> 1953293 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62522_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1953298 INFO  (OverseerStateUpdate-97461928759263236-127.0.0.1:62522_-n_0000000000) [n:127.0.0.1:62522_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1954315 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1954326 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1954378 WARN  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1954380 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1954407 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1954408 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/control-001/cores/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/control-001/cores/collection1/data/]
   [junit4]   2> 1954408 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@10b647c2
   [junit4]   2> 1954409 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=72.546875, floorSegmentMB=1.21875, forceMergeDeletesPctAllowed=10.314984739534411, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1954413 WARN  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1954465 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1954466 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1954467 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1954467 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1954467 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18186766922857434]
   [junit4]   2> 1954468 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@63054e86[collection1] main]
   [junit4]   2> 1954470 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1954470 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1954470 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1954471 INFO  (searcherExecutor-7283-thread-1-processing-n:127.0.0.1:62522_ x:collection1 c:control_collection) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@63054e86[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1954472 INFO  (coreLoadExecutor-7282-thread-1-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559390862905966592
   [junit4]   2> 1954482 INFO  (coreZkRegister-7275-thread-1-processing-n:127.0.0.1:62522_ x:collection1 c:control_collection) [n:127.0.0.1:62522_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1954482 INFO  (coreZkRegister-7275-thread-1-processing-n:127.0.0.1:62522_ x:collection1 c:control_collection) [n:127.0.0.1:62522_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1954482 INFO  (coreZkRegister-7275-thread-1-processing-n:127.0.0.1:62522_ x:collection1 c:control_collection) [n:127.0.0.1:62522_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:62522/collection1/
   [junit4]   2> 1954483 INFO  (coreZkRegister-7275-thread-1-processing-n:127.0.0.1:62522_ x:collection1 c:control_collection) [n:127.0.0.1:62522_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1954483 INFO  (coreZkRegister-7275-thread-1-processing-n:127.0.0.1:62522_ x:collection1 c:control_collection) [n:127.0.0.1:62522_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:62522/collection1/ has no replicas
   [junit4]   2> 1954490 INFO  (coreZkRegister-7275-thread-1-processing-n:127.0.0.1:62522_ x:collection1 c:control_collection) [n:127.0.0.1:62522_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:62522/collection1/ shard1
   [junit4]   2> 1954563 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1954565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:62519/solr ready
   [junit4]   2> 1954565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1954565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1954603 INFO  (coreZkRegister-7275-thread-1-processing-n:127.0.0.1:62522_ x:collection1 c:control_collection) [n:127.0.0.1:62522_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1954864 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-1-001/cores/collection1
   [junit4]   2> 1954865 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-1-001
   [junit4]   2> 1954865 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1954867 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d552995{/,null,AVAILABLE}
   [junit4]   2> 1954868 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5018d29d{HTTP/1.1,[http/1.1]}{127.0.0.1:62528}
   [junit4]   2> 1954868 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.Server Started @1960855ms
   [junit4]   2> 1954868 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=62528, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-1-001/cores}
   [junit4]   2> 1954868 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1954869 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1954869 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1954869 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1954869 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-15T09:30:15.589Z
   [junit4]   2> 1954873 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1954873 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-1-001/solr.xml
   [junit4]   2> 1954889 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:62519/solr
   [junit4]   2> 1954905 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62528_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1954911 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62528_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1954914 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62528_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:62528_
   [junit4]   2> 1954917 INFO  (zkCallback-3091-thread-2-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1954917 INFO  (zkCallback-3095-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1954918 INFO  (zkCallback-3101-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1954997 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62528_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-1-001/cores
   [junit4]   2> 1954997 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62528_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1955001 INFO  (OverseerStateUpdate-97461928759263236-127.0.0.1:62522_-n_0000000000) [n:127.0.0.1:62522_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1955115 INFO  (zkCallback-3101-thread-2-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1956021 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1956031 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1956082 WARN  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1956083 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1956110 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1956110 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-1-001/cores/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 1956110 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@10b647c2
   [junit4]   2> 1956112 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=72.546875, floorSegmentMB=1.21875, forceMergeDeletesPctAllowed=10.314984739534411, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1956116 WARN  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1956169 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1956169 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1956170 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1956170 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1956170 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18186766922857434]
   [junit4]   2> 1956171 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6c0a570a[collection1] main]
   [junit4]   2> 1956173 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1956173 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1956173 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1956174 INFO  (searcherExecutor-7294-thread-1-processing-n:127.0.0.1:62528_ x:collection1 c:collection1) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6c0a570a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1956175 INFO  (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559390864691691520
   [junit4]   2> 1956184 INFO  (coreZkRegister-7288-thread-1-processing-n:127.0.0.1:62528_ x:collection1 c:collection1) [n:127.0.0.1:62528_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1956184 INFO  (coreZkRegister-7288-thread-1-processing-n:127.0.0.1:62528_ x:collection1 c:collection1) [n:127.0.0.1:62528_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1956184 INFO  (coreZkRegister-7288-thread-1-processing-n:127.0.0.1:62528_ x:collection1 c:collection1) [n:127.0.0.1:62528_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:62528/collection1/
   [junit4]   2> 1956185 INFO  (coreZkRegister-7288-thread-1-processing-n:127.0.0.1:62528_ x:collection1 c:collection1) [n:127.0.0.1:62528_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1956185 INFO  (coreZkRegister-7288-thread-1-processing-n:127.0.0.1:62528_ x:collection1 c:collection1) [n:127.0.0.1:62528_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:62528/collection1/ has no replicas
   [junit4]   2> 1956192 INFO  (coreZkRegister-7288-thread-1-processing-n:127.0.0.1:62528_ x:collection1 c:collection1) [n:127.0.0.1:62528_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:62528/collection1/ shard2
   [junit4]   2> 1956307 INFO  (zkCallback-3101-thread-2-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1956366 INFO  (coreZkRegister-7288-thread-1-processing-n:127.0.0.1:62528_ x:collection1 c:collection1) [n:127.0.0.1:62528_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1956478 INFO  (zkCallback-3101-thread-2-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1956947 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-2-001/cores/collection1
   [junit4]   2> 1956948 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-2-001
   [junit4]   2> 1956949 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1956951 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@13c87c16{/,null,AVAILABLE}
   [junit4]   2> 1956951 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7a713323{HTTP/1.1,[http/1.1]}{127.0.0.1:62532}
   [junit4]   2> 1956951 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.e.j.s.Server Started @1962938ms
   [junit4]   2> 1956951 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=62532, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-2-001/cores}
   [junit4]   2> 1956952 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1956952 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1956952 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1956952 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1956953 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-15T09:30:17.673Z
   [junit4]   2> 1956957 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1956957 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_D08C0DB04A5A4B25-001/shard-2-001/solr.xml
   [junit4]   2> 1956982 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:62519/solr
   [junit4]   2> 1956999 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62532_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1957004 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62532_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1957009 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[D08C0DB04A5A4B25]) [n:127.0.0.1:62532_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:62532_
   [junit4]   2> 1957012 INFO  (zkCallback-3101-thread-2-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1957012 INFO  (zkCallback-3091-thread-3-processing-n:127.0.0.1:62522_) [n:127.0.0.1:62522_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1957012 INFO  (zkCallback-3095-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1957013 INFO  (zkCallback-3107-thread-1-processing-n:127.0.0.1:62532_) [n:127.0.0.1:62532_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1957124 INFO  (zkCallback-3101-thread-2-processing-n:127.0.0.1:62528_) [n:127.0.0.1:62528_    ] o.

[...truncated too long message...]

st, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, ZkCLITest, TestDistributedGrouping, TestReload, TestSolr4Spatial, SpellCheckComponentTest, SolrCmdDistributorTest, BadIndexSchemaTest, TestSort, TestFiltering, BasicFunctionalityTest, TestLazyCores, HighlighterTest, SignatureUpdateProcessorFactoryTest, SolrRequestParserTest, SuggesterTest, TestTrie, TestCSVLoader, PolyFieldTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, TestUpdate, TestRemoteStreaming, XsltUpdateRequestHandlerTest, DebugComponentTest, TestSurroundQueryParser, LukeRequestHandlerTest, PathHierarchyTokenizerFactoryTest, TestSolrQueryParser, SolrPluginUtilsTest, TestCollationField, MBeansHandlerTest, TestAnalyzedSuggestions, JsonLoaderTest, SOLR749Test, CopyFieldTest, BadComponentTest, TestSolrDeletionPolicy2, TestSearchPerf, TestConfig, TestSolrCoreProperties, DirectSolrConnectionTest, TestLuceneMatchVersion, TestCodecSupport, TestLMJelinekMercerSimilarityFactory, TestFastOutputStream, TestSystemIdResolver, DateFieldTest, ClusterStateTest, TestLRUCache, SliceStateTest, UUIDFieldTest, CursorPagingTest, CdcrReplicationHandlerTest, CdcrVersionReplicationTest, CollectionsAPIAsyncDistributedZkTest, ConcurrentDeleteAndCreateCollectionTest, ConnectionManagerTest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest, DeleteStatusTest, DistributedQueueTest, DocValuesNotIndexedTest, LeaderElectionContextKeyTest, LeaderFailureAfterFreshStartTest, MigrateRouteKeyTest, MultiThreadedOCPTest, RecoveryAfterSoftCommitTest, ReplicationFactorTest, RestartWhileUpdatingTest, SSLMigrationTest, ShardSplitTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ShardSplitTest -Dtests.seed=D08C0DB04A5A4B25 -Dtests.slow=true -Dtests.locale=ar-DZ -Dtests.timezone=Europe/Zurich -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | ShardSplitTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest:
   [junit4]    >    1) Thread[id=19862, name=TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D08C0DB04A5A4B25]-EventThread, state=WAITING, group=TGRP-ShardSplitTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    >    2) Thread[id=19853, name=Thread-3634, state=WAITING, group=TGRP-ShardSplitTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1249)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1323)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]    >    3) Thread[id=19861, name=TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D08C0DB04A5A4B25]-SendThread(127.0.0.1:64006), state=RUNNABLE, group=TGRP-ShardSplitTest]
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   [junit4]    >         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]    >    4) Thread[id=19867, name=Thread-3637, state=WAITING, group=TGRP-ShardSplitTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1249)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1323)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]    >    5) Thread[id=19871, name=ProcessThread(sid:0 cport:-1):, state=WAITING, group=TGRP-ShardSplitTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)
   [junit4]    >    6) Thread[id=19857, name=ProcessThread(sid:0 cport:-1):, state=WAITING, group=TGRP-ShardSplitTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)
   [junit4]    >    7) Thread[id=19855, name=SessionTracker, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]    >    8) Thread[id=19869, name=SessionTracker, state=TIMED_WAITING, group=TGRP-ShardSplitTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]    >    9) Thread[id=19868, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   [junit4]    >         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   10) Thread[id=19854, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   [junit4]    >         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([D08C0DB04A5A4B25]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=19861, name=TEST-ShardSplitTest.testSplitWithChaosMonkey-seed#[D08C0DB04A5A4B25]-SendThread(127.0.0.1:64006), state=TIMED_WAITING, group=TGRP-ShardSplitTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    2) Thread[id=19868, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-ShardSplitTest]
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   [junit4]    >         at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > at __randomizedtesting.SeedInfo.seed([D08C0DB04A5A4B25]:0)
   [junit4] Completed [507/694 (4!)] on J1 in 131.35s, 5 tests, 7 errors <<< FAILURES!

[...truncated 63184 lines...]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3835 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3835/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

4 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter

Error Message:
Collection not found: withShardField

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: withShardField
        at __randomizedtesting.SeedInfo.seed([723A1655EA4FAD35:276AFEC746B662C5]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1376)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1072)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:232)
        at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.PeerSyncReplicationTest.test

Error Message:
timeout waiting to see all nodes active

Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
        at __randomizedtesting.SeedInfo.seed([723A1655EA4FAD35:FA6E298F44B3C0CD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
        at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
        at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
        at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud

Error Message:
Error from server at http://127.0.0.1:64036/solr: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/admin/collections. Reason: <pre>    Can not find: /solr/admin/collections</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/> </body> </html>

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:64036/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Can not find: /solr/admin/collections</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/>
</body>
</html>

        at __randomizedtesting.SeedInfo.seed([723A1655EA4FAD35]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:439)
        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:391)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1358)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1109)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
        at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorRandomCloud.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:847)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestMiniSolrCloudCluster.testCollectionCreateSearchDelete

Error Message:
Could not find collection:testcollection

Stack Trace:
java.lang.AssertionError: Could not find collection:testcollection
        at __randomizedtesting.SeedInfo.seed([723A1655EA4FAD35:D1C0B8F06DA74790]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
        at org.apache.solr.cloud.TestMiniSolrCloudCluster.testCollectionCreateSearchDelete(TestMiniSolrCloudCluster.java:165)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12387 lines...]
   [junit4] Suite: org.apache.solr.cloud.CustomCollectionTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/init-core-data-001
   [junit4]   2> 2507131 INFO  (SUITE-CustomCollectionTest-seed#[723A1655EA4FAD35]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 2507135 INFO  (SUITE-CustomCollectionTest-seed#[723A1655EA4FAD35]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2507136 INFO  (SUITE-CustomCollectionTest-seed#[723A1655EA4FAD35]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001
   [junit4]   2> 2507137 INFO  (SUITE-CustomCollectionTest-seed#[723A1655EA4FAD35]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2507137 INFO  (Thread-3873) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2507137 INFO  (Thread-3873) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2507237 INFO  (SUITE-CustomCollectionTest-seed#[723A1655EA4FAD35]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:61497
   [junit4]   2> 2507263 INFO  (jetty-launcher-3237-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2507263 INFO  (jetty-launcher-3237-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2507263 INFO  (jetty-launcher-3237-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2507264 INFO  (jetty-launcher-3237-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2507265 INFO  (jetty-launcher-3237-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@68d6dfab{/solr,null,AVAILABLE}
   [junit4]   2> 2507267 INFO  (jetty-launcher-3237-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@11fb940f{/solr,null,AVAILABLE}
   [junit4]   2> 2507268 INFO  (jetty-launcher-3237-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6da15cac{SSL,[ssl, http/1.1]}{127.0.0.1:61499}
   [junit4]   2> 2507268 INFO  (jetty-launcher-3237-thread-2) [    ] o.e.j.s.Server Started @2512114ms
   [junit4]   2> 2507268 INFO  (jetty-launcher-3237-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61499}
   [junit4]   2> 2507268 ERROR (jetty-launcher-3237-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2507268 INFO  (jetty-launcher-3237-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2507269 INFO  (jetty-launcher-3237-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2507269 INFO  (jetty-launcher-3237-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2507269 INFO  (jetty-launcher-3237-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-15T19:21:42.372Z
   [junit4]   2> 2507271 INFO  (jetty-launcher-3237-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7685c36a{/solr,null,AVAILABLE}
   [junit4]   2> 2507271 INFO  (jetty-launcher-3237-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6475b9ee{SSL,[ssl, http/1.1]}{127.0.0.1:61500}
   [junit4]   2> 2507271 INFO  (jetty-launcher-3237-thread-3) [    ] o.e.j.s.Server Started @2512118ms
   [junit4]   2> 2507271 INFO  (jetty-launcher-3237-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61500}
   [junit4]   2> 2507272 ERROR (jetty-launcher-3237-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2507272 INFO  (jetty-launcher-3237-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@684f9672{SSL,[ssl, http/1.1]}{127.0.0.1:61502}
   [junit4]   2> 2507272 INFO  (jetty-launcher-3237-thread-4) [    ] o.e.j.s.Server Started @2512118ms
   [junit4]   2> 2507272 INFO  (jetty-launcher-3237-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61502}
   [junit4]   2> 2507272 INFO  (jetty-launcher-3237-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2507272 ERROR (jetty-launcher-3237-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2507272 INFO  (jetty-launcher-3237-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2507272 INFO  (jetty-launcher-3237-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2507272 INFO  (jetty-launcher-3237-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-15T19:21:42.375Z
   [junit4]   2> 2507272 INFO  (jetty-launcher-3237-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2507273 INFO  (jetty-launcher-3237-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2507273 INFO  (jetty-launcher-3237-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2507273 INFO  (jetty-launcher-3237-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-15T19:21:42.376Z
   [junit4]   2> 2507273 INFO  (jetty-launcher-3237-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5e5f1b3a{/solr,null,AVAILABLE}
   [junit4]   2> 2507274 INFO  (jetty-launcher-3237-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7d64d577{SSL,[ssl, http/1.1]}{127.0.0.1:61503}
   [junit4]   2> 2507274 INFO  (jetty-launcher-3237-thread-1) [    ] o.e.j.s.Server Started @2512120ms
   [junit4]   2> 2507274 INFO  (jetty-launcher-3237-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61503}
   [junit4]   2> 2507274 ERROR (jetty-launcher-3237-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2507274 INFO  (jetty-launcher-3237-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2507274 INFO  (jetty-launcher-3237-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2507274 INFO  (jetty-launcher-3237-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2507275 INFO  (jetty-launcher-3237-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-15T19:21:42.377Z
   [junit4]   2> 2507276 INFO  (jetty-launcher-3237-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2507279 INFO  (jetty-launcher-3237-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2507279 INFO  (jetty-launcher-3237-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2507280 INFO  (jetty-launcher-3237-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2507303 INFO  (jetty-launcher-3237-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61497/solr
   [junit4]   2> 2507304 INFO  (jetty-launcher-3237-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61497/solr
   [junit4]   2> 2507307 INFO  (jetty-launcher-3237-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61497/solr
   [junit4]   2> 2507348 INFO  (jetty-launcher-3237-thread-4) [n:127.0.0.1:61502_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2507348 INFO  (jetty-launcher-3237-thread-1) [n:127.0.0.1:61503_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2507349 INFO  (jetty-launcher-3237-thread-1) [n:127.0.0.1:61503_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:61503_solr
   [junit4]   2> 2507349 INFO  (jetty-launcher-3237-thread-3) [n:127.0.0.1:61500_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2507350 INFO  (jetty-launcher-3237-thread-4) [n:127.0.0.1:61502_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61502_solr
   [junit4]   2> 2507351 INFO  (jetty-launcher-3237-thread-1) [n:127.0.0.1:61503_solr    ] o.a.s.c.Overseer Overseer (id=97464254610997257-127.0.0.1:61503_solr-n_0000000000) starting
   [junit4]   2> 2507351 INFO  (jetty-launcher-3237-thread-3) [n:127.0.0.1:61500_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61500_solr
   [junit4]   2> 2507354 INFO  (zkCallback-3254-thread-1-processing-n:127.0.0.1:61500_solr) [n:127.0.0.1:61500_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2507354 INFO  (zkCallback-3253-thread-1-processing-n:127.0.0.1:61503_solr) [n:127.0.0.1:61503_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2507354 INFO  (zkCallback-3251-thread-1-processing-n:127.0.0.1:61502_solr) [n:127.0.0.1:61502_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2507354 INFO  (jetty-launcher-3237-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61497/solr
   [junit4]   2> 2507356 INFO  (zkCallback-3253-thread-1-processing-n:127.0.0.1:61503_solr) [n:127.0.0.1:61503_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2507357 INFO  (zkCallback-3254-thread-1-processing-n:127.0.0.1:61500_solr) [n:127.0.0.1:61500_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2507357 INFO  (zkCallback-3251-thread-1-processing-n:127.0.0.1:61502_solr) [n:127.0.0.1:61502_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2507367 INFO  (jetty-launcher-3237-thread-1) [n:127.0.0.1:61503_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61503_solr
   [junit4]   2> 2507371 INFO  (zkCallback-3254-thread-1-processing-n:127.0.0.1:61500_solr) [n:127.0.0.1:61500_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2507371 INFO  (zkCallback-3251-thread-1-processing-n:127.0.0.1:61502_solr) [n:127.0.0.1:61502_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2507371 INFO  (zkCallback-3253-thread-1-processing-n:127.0.0.1:61503_solr) [n:127.0.0.1:61503_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2507396 INFO  (jetty-launcher-3237-thread-2) [n:127.0.0.1:61499_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2507400 INFO  (jetty-launcher-3237-thread-2) [n:127.0.0.1:61499_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2507403 INFO  (jetty-launcher-3237-thread-2) [n:127.0.0.1:61499_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61499_solr
   [junit4]   2> 2507407 INFO  (zkCallback-3253-thread-2-processing-n:127.0.0.1:61503_solr) [n:127.0.0.1:61503_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2507407 INFO  (zkCallback-3251-thread-1-processing-n:127.0.0.1:61502_solr) [n:127.0.0.1:61502_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2507407 INFO  (zkCallback-3254-thread-1-processing-n:127.0.0.1:61500_solr) [n:127.0.0.1:61500_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2507409 INFO  (zkCallback-3261-thread-1-processing-n:127.0.0.1:61499_solr) [n:127.0.0.1:61499_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2507485 INFO  (jetty-launcher-3237-thread-2) [n:127.0.0.1:61499_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node2/.
   [junit4]   2> 2507487 INFO  (jetty-launcher-3237-thread-1) [n:127.0.0.1:61503_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node1/.
   [junit4]   2> 2507504 INFO  (jetty-launcher-3237-thread-4) [n:127.0.0.1:61502_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node4/.
   [junit4]   2> 2507701 INFO  (jetty-launcher-3237-thread-3) [n:127.0.0.1:61500_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node3/.
   [junit4]   2> 2509475 INFO  (SUITE-CustomCollectionTest-seed#[723A1655EA4FAD35]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2509477 INFO  (SUITE-CustomCollectionTest-seed#[723A1655EA4FAD35]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:61497/solr ready
   [junit4]   2> 2509517 INFO  (TEST-CustomCollectionTest.testCustomCollectionsAPI-seed#[723A1655EA4FAD35]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCustomCollectionsAPI
   [junit4]   2> 2509519 INFO  (TEST-CustomCollectionTest.testCustomCollectionsAPI-seed#[723A1655EA4FAD35]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2509542 INFO  (qtp124828377-18572) [n:127.0.0.1:61502_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=a,b,c&replicationFactor=5&collection.configName=conf&maxShardsPerNode=6&name=implicitcoll&router.name=implicit&action=CREATE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2509545 INFO  (OverseerThreadFactory-8825-thread-1-processing-n:127.0.0.1:61503_solr) [n:127.0.0.1:61503_solr    ] o.a.s.c.CreateCollectionCmd Create collection implicitcoll
   [junit4]   2> 2509546 WARN  (OverseerThreadFactory-8825-thread-1-processing-n:127.0.0.1:61503_solr) [n:127.0.0.1:61503_solr    ] o.a.s.c.CreateCollectionCmd Specified replicationFactor of 5 on collection implicitcoll is higher than or equal to the number of Solr instances currently live or live and part of your createNodeSet(4). It's unusual to run two replica of the same slice on the same Solr-instance.
   [junit4]   2> 2509719 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_a_replica1&action=CREATE&numShards=3&collection=implicitcoll&shard=a&wt=javabin&version=2
   [junit4]   2> 2509756 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_a_replica2&action=CREATE&numShards=3&collection=implicitcoll&shard=a&wt=javabin&version=2
   [junit4]   2> 2509757 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_c_replica4&action=CREATE&numShards=3&collection=implicitcoll&shard=c&wt=javabin&version=2
   [junit4]   2> 2509758 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_b_replica1&action=CREATE&numShards=3&collection=implicitcoll&shard=b&wt=javabin&version=2
   [junit4]   2> 2509758 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_b_replica5&action=CREATE&numShards=3&collection=implicitcoll&shard=b&wt=javabin&version=2
   [junit4]   2> 2509782 INFO  (qtp697516828-18551) [n:127.0.0.1:61500_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_c_replica1&action=CREATE&numShards=3&collection=implicitcoll&shard=c&wt=javabin&version=2
   [junit4]   2> 2509783 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_a_replica5&action=CREATE&numShards=3&collection=implicitcoll&shard=a&wt=javabin&version=2
   [junit4]   2> 2509787 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_b_replica3&action=CREATE&numShards=3&collection=implicitcoll&shard=b&wt=javabin&version=2
   [junit4]   2> 2509788 INFO  (qtp697516828-18553) [n:127.0.0.1:61500_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_a_replica3&action=CREATE&numShards=3&collection=implicitcoll&shard=a&wt=javabin&version=2
   [junit4]   2> 2509789 INFO  (qtp697516828-18552) [n:127.0.0.1:61500_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_c_replica5&action=CREATE&numShards=3&collection=implicitcoll&shard=c&wt=javabin&version=2
   [junit4]   2> 2509793 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_b_replica4&action=CREATE&numShards=3&collection=implicitcoll&shard=b&wt=javabin&version=2
   [junit4]   2> 2509793 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_c_replica3&action=CREATE&numShards=3&collection=implicitcoll&shard=c&wt=javabin&version=2
   [junit4]   2> 2509796 INFO  (qtp2094182752-18562) [n:127.0.0.1:61503_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_a_replica4&action=CREATE&numShards=3&collection=implicitcoll&shard=a&wt=javabin&version=2
   [junit4]   2> 2509799 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_c_replica2&action=CREATE&numShards=3&collection=implicitcoll&shard=c&wt=javabin&version=2
   [junit4]   2> 2509800 INFO  (qtp697516828-18555) [n:127.0.0.1:61500_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=implicitcoll_b_replica2&action=CREATE&numShards=3&collection=implicitcoll&shard=b&wt=javabin&version=2
   [junit4]   2> 2509929 INFO  (zkCallback-3251-thread-1-processing-n:127.0.0.1:61502_solr) [n:127.0.0.1:61502_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/implicitcoll/state.json] for collection [implicitcoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2509929 INFO  (zkCallback-3261-thread-1-processing-n:127.0.0.1:61499_solr) [n:127.0.0.1:61499_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/implicitcoll/state.json] for collection [implicitcoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2509929 INFO  (zkCallback-3254-thread-1-processing-n:127.0.0.1:61500_solr) [n:127.0.0.1:61500_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/implicitcoll/state.json] for collection [implicitcoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2509929 INFO  (zkCallback-3253-thread-3-processing-n:127.0.0.1:61503_solr) [n:127.0.0.1:61503_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/implicitcoll/state.json] for collection [implicitcoll] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2510792 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510792 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510794 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510803 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510812 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510825 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.s.IndexSchema [implicitcoll_a_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2510828 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.s.IndexSchema [implicitcoll_b_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2510838 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510842 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510858 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.s.IndexSchema [implicitcoll_c_replica4] Schema name=cloud-dynamic
   [junit4]   2> 2510859 WARN  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.s.IndexSchema [implicitcoll_a_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2510860 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2510863 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510864 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.s.IndexSchema [implicitcoll_a_replica2] Schema name=cloud-dynamic
   [junit4]   2> 2510878 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510878 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_a_replica1' using configuration from collection implicitcoll
   [junit4]   2> 2510879 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.c.SolrCore [[implicitcoll_a_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node2/implicitcoll_a_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node2/./implicitcoll_a_replica1/data/]
   [junit4]   2> 2510888 INFO  (qtp2094182752-18562) [n:127.0.0.1:61503_solr c:implicitcoll s:a  x:implicitcoll_a_replica4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510888 INFO  (qtp697516828-18552) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510892 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.s.IndexSchema [implicitcoll_b_replica4] Schema name=cloud-dynamic
   [junit4]   2> 2510905 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.s.IndexSchema [implicitcoll_a_replica5] Schema name=cloud-dynamic
   [junit4]   2> 2510927 INFO  (qtp697516828-18553) [n:127.0.0.1:61500_solr c:implicitcoll s:a  x:implicitcoll_a_replica3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510928 INFO  (qtp697516828-18551) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510936 INFO  (qtp697516828-18555) [n:127.0.0.1:61500_solr c:implicitcoll s:b  x:implicitcoll_b_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510940 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2510962 INFO  (qtp697516828-18553) [n:127.0.0.1:61500_solr c:implicitcoll s:a  x:implicitcoll_a_replica3] o.a.s.s.IndexSchema [implicitcoll_a_replica3] Schema name=cloud-dynamic
   [junit4]   2> 2510964 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.s.IndexSchema [implicitcoll_b_replica5] Schema name=cloud-dynamic
   [junit4]   2> 2510965 INFO  (qtp697516828-18555) [n:127.0.0.1:61500_solr c:implicitcoll s:b  x:implicitcoll_b_replica2] o.a.s.s.IndexSchema [implicitcoll_b_replica2] Schema name=cloud-dynamic
   [junit4]   2> 2510966 INFO  (qtp697516828-18551) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica1] o.a.s.s.IndexSchema [implicitcoll_c_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2510971 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.s.IndexSchema [implicitcoll_c_replica2] Schema name=cloud-dynamic
   [junit4]   2> 2510984 WARN  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.s.IndexSchema [implicitcoll_a_replica5] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2510985 WARN  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.s.IndexSchema [implicitcoll_a_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2510985 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2510985 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2510988 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_a_replica5' using configuration from collection implicitcoll
   [junit4]   2> 2510989 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.c.SolrCore [[implicitcoll_a_replica5] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node2/implicitcoll_a_replica5], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node2/./implicitcoll_a_replica5/data/]
   [junit4]   2> 2510988 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.s.IndexSchema [implicitcoll_c_replica3] Schema name=cloud-dynamic
   [junit4]   2> 2510987 INFO  (qtp2094182752-18562) [n:127.0.0.1:61503_solr c:implicitcoll s:a  x:implicitcoll_a_replica4] o.a.s.s.IndexSchema [implicitcoll_a_replica4] Schema name=cloud-dynamic
   [junit4]   2> 2510986 INFO  (qtp697516828-18552) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica5] o.a.s.s.IndexSchema [implicitcoll_c_replica5] Schema name=cloud-dynamic
   [junit4]   2> 2510986 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.s.IndexSchema [implicitcoll_b_replica3] Schema name=cloud-dynamic
   [junit4]   2> 2511020 WARN  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.s.IndexSchema [implicitcoll_c_replica4] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511021 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511045 WARN  (qtp697516828-18555) [n:127.0.0.1:61500_solr c:implicitcoll s:b  x:implicitcoll_b_replica2] o.a.s.s.IndexSchema [implicitcoll_b_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511046 INFO  (qtp697516828-18555) [n:127.0.0.1:61500_solr c:implicitcoll s:b  x:implicitcoll_b_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511058 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_a_replica2' using configuration from collection implicitcoll
   [junit4]   2> 2511069 INFO  (qtp697516828-18555) [n:127.0.0.1:61500_solr c:implicitcoll s:b  x:implicitcoll_b_replica2] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_b_replica2' using configuration from collection implicitcoll
   [junit4]   2> 2511069 INFO  (qtp697516828-18555) [n:127.0.0.1:61500_solr c:implicitcoll s:b  x:implicitcoll_b_replica2] o.a.s.c.SolrCore [[implicitcoll_b_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node3/implicitcoll_b_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node3/./implicitcoll_b_replica2/data/]
   [junit4]   2> 2511071 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_c_replica4' using configuration from collection implicitcoll
   [junit4]   2> 2511071 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.c.SolrCore [[implicitcoll_c_replica4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node4/implicitcoll_c_replica4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node4/./implicitcoll_c_replica4/data/]
   [junit4]   2> 2511075 WARN  (qtp697516828-18551) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica1] o.a.s.s.IndexSchema [implicitcoll_c_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511079 WARN  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.s.IndexSchema [implicitcoll_b_replica4] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511080 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511082 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_b_replica4' using configuration from collection implicitcoll
   [junit4]   2> 2511082 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.c.SolrCore [[implicitcoll_b_replica4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node2/implicitcoll_b_replica4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node2/./implicitcoll_b_replica4/data/]
   [junit4]   2> 2511083 WARN  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.s.IndexSchema [implicitcoll_c_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511084 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511085 INFO  (qtp697516828-18551) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511085 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_c_replica2' using configuration from collection implicitcoll
   [junit4]   2> 2511085 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.c.SolrCore [[implicitcoll_c_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node1/implicitcoll_c_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node1/./implicitcoll_c_replica2/data/]
   [junit4]   2> 2511086 INFO  (qtp697516828-18551) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica1] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_c_replica1' using configuration from collection implicitcoll
   [junit4]   2> 2511086 INFO  (qtp697516828-18551) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica1] o.a.s.c.SolrCore [[implicitcoll_c_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node3/implicitcoll_c_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node3/./implicitcoll_c_replica1/data/]
   [junit4]   2> 2511088 WARN  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.s.IndexSchema [implicitcoll_b_replica5] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511089 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511090 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_b_replica5' using configuration from collection implicitcoll
   [junit4]   2> 2511090 WARN  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.s.IndexSchema [implicitcoll_b_replica3] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511090 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.c.SolrCore [[implicitcoll_b_replica5] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node4/implicitcoll_b_replica5], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node4/./implicitcoll_b_replica5/data/]
   [junit4]   2> 2511091 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511102 WARN  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.s.IndexSchema [implicitcoll_b_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511102 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.c.SolrCore [[implicitcoll_a_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node4/implicitcoll_a_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node4/./implicitcoll_a_replica2/data/]
   [junit4]   2> 2511103 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511102 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_b_replica3' using configuration from collection implicitcoll
   [junit4]   2> 2511104 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.c.SolrCore [[implicitcoll_b_replica3] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node1/implicitcoll_b_replica3], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node1/./implicitcoll_b_replica3/data/]
   [junit4]   2> 2511104 WARN  (qtp697516828-18553) [n:127.0.0.1:61500_solr c:implicitcoll s:a  x:implicitcoll_a_replica3] o.a.s.s.IndexSchema [implicitcoll_a_replica3] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511105 INFO  (qtp697516828-18553) [n:127.0.0.1:61500_solr c:implicitcoll s:a  x:implicitcoll_a_replica3] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511106 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_b_replica1' using configuration from collection implicitcoll
   [junit4]   2> 2511106 INFO  (qtp697516828-18553) [n:127.0.0.1:61500_solr c:implicitcoll s:a  x:implicitcoll_a_replica3] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_a_replica3' using configuration from collection implicitcoll
   [junit4]   2> 2511106 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.c.SolrCore [[implicitcoll_b_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node4/implicitcoll_b_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node4/./implicitcoll_b_replica1/data/]
   [junit4]   2> 2511106 INFO  (qtp697516828-18553) [n:127.0.0.1:61500_solr c:implicitcoll s:a  x:implicitcoll_a_replica3] o.a.s.c.SolrCore [[implicitcoll_a_replica3] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node3/implicitcoll_a_replica3], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node3/./implicitcoll_a_replica3/data/]
   [junit4]   2> 2511112 WARN  (qtp697516828-18552) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica5] o.a.s.s.IndexSchema [implicitcoll_c_replica5] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511113 WARN  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.s.IndexSchema [implicitcoll_c_replica3] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511113 INFO  (qtp697516828-18552) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica5] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511113 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511114 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_c_replica3' using configuration from collection implicitcoll
   [junit4]   2> 2511115 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.c.SolrCore [[implicitcoll_c_replica3] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node2/implicitcoll_c_replica3], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node2/./implicitcoll_c_replica3/data/]
   [junit4]   2> 2511117 INFO  (qtp697516828-18552) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica5] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_c_replica5' using configuration from collection implicitcoll
   [junit4]   2> 2511117 INFO  (qtp697516828-18552) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica5] o.a.s.c.SolrCore [[implicitcoll_c_replica5] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node3/implicitcoll_c_replica5], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node3/./implicitcoll_c_replica5/data/]
   [junit4]   2> 2511121 WARN  (qtp2094182752-18562) [n:127.0.0.1:61503_solr c:implicitcoll s:a  x:implicitcoll_a_replica4] o.a.s.s.IndexSchema [implicitcoll_a_replica4] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2511121 INFO  (qtp2094182752-18562) [n:127.0.0.1:61503_solr c:implicitcoll s:a  x:implicitcoll_a_replica4] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2511123 INFO  (qtp2094182752-18562) [n:127.0.0.1:61503_solr c:implicitcoll s:a  x:implicitcoll_a_replica4] o.a.s.c.CoreContainer Creating SolrCore 'implicitcoll_a_replica4' using configuration from collection implicitcoll
   [junit4]   2> 2511123 INFO  (qtp2094182752-18562) [n:127.0.0.1:61503_solr c:implicitcoll s:a  x:implicitcoll_a_replica4] o.a.s.c.SolrCore [[implicitcoll_a_replica4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node1/implicitcoll_a_replica4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_723A1655EA4FAD35-001/tempDir-001/node1/./implicitcoll_a_replica4/data/]
   [junit4]   2> 2511276 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511276 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511278 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511278 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511282 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4a62b1ef[implicitcoll_a_replica1] main]
   [junit4]   2> 2511309 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511309 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511309 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2511309 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2511310 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2511310 INFO  (qtp620300327-18543) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559428078148517888
   [junit4]   2> 2511310 INFO  (searcherExecutor-8834-thread-1-processing-n:127.0.0.1:61499_solr x:implicitcoll_a_replica1 s:a c:implicitcoll) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica1] o.a.s.c.SolrCore [implicitcoll_a_replica1] Registered new searcher Searcher@4a62b1ef[implicitcoll_a_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2511310 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511310 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511312 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511312 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511312 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511312 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511321 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511321 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511323 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511323 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511325 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511325 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511327 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511327 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511332 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511332 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511334 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511334 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511341 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@2f308f0b[implicitcoll_c_replica2] main]
   [junit4]   2> 2511341 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.s.SolrIndexSearcher Opening [Searcher@6e63c497[implicitcoll_b_replica5] main]
   [junit4]   2> 2511346 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2511349 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2511349 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2511349 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559428078189412352
   [junit4]   2> 2511349 INFO  (searcherExecutor-8840-thread-1-processing-n:127.0.0.1:61503_solr x:implicitcoll_c_replica2 s:c c:implicitcoll) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.c.SolrCore [implicitcoll_c_replica2] Registered new searcher Searcher@2f308f0b[implicitcoll_c_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2511364 INFO  (qtp2094182752-18560) [n:127.0.0.1:61503_solr c:implicitcoll s:c  x:implicitcoll_c_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard c: total=5 found=1 timeoutin=9999ms
   [junit4]   2> 2511367 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2511370 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2511370 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2511370 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.s.SolrIndexSearcher Opening [Searcher@3498ce8c[implicitcoll_a_replica5] main]
   [junit4]   2> 2511370 INFO  (qtp124828377-18576) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559428078211432448
   [junit4]   2> 2511370 INFO  (searcherExecutor-8842-thread-1-processing-n:127.0.0.1:61502_solr x:implicitcoll_b_replica5 s:b c:implicitcoll) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica5] o.a.s.c.SolrCore [implicitcoll_b_replica5] Registered new searcher Searcher@6e63c497[implicitcoll_b_replica5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2511374 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.s.SolrIndexSearcher Opening [Searcher@6d69f2b[implicitcoll_c_replica4] main]
   [junit4]   2> 2511378 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@406fb8b[implicitcoll_a_replica2] main]
   [junit4]   2> 2511381 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511381 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511383 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511383 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511385 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511385 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511389 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.s.SolrIndexSearcher Opening [Searcher@4fa06f8e[implicitcoll_b_replica4] main]
   [junit4]   2> 2511390 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511391 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511392 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7aa85a4b[implicitcoll_b_replica1] main]
   [junit4]   2> 2511395 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511395 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511397 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2511397 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511397 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511397 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2511397 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2511397 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2511398 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2511398 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2511399 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2511399 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559428078241841152
   [junit4]   2> 2511399 INFO  (searcherExecutor-8835-thread-1-processing-n:127.0.0.1:61499_solr x:implicitcoll_a_replica5 s:a c:implicitcoll) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.c.SolrCore [implicitcoll_a_replica5] Registered new searcher Searcher@3498ce8c[implicitcoll_a_replica5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2511399 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559428078241841152
   [junit4]   2> 2511399 INFO  (searcherExecutor-8844-thread-1-processing-n:127.0.0.1:61502_solr x:implicitcoll_b_replica1 s:b c:implicitcoll) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.c.SolrCore [implicitcoll_b_replica1] Registered new searcher Searcher@7aa85a4b[implicitcoll_b_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2511403 INFO  (qtp2094182752-18562) [n:127.0.0.1:61503_solr c:implicitcoll s:a  x:implicitcoll_a_replica4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511403 INFO  (qtp2094182752-18562) [n:127.0.0.1:61503_solr c:implicitcoll s:a  x:implicitcoll_a_replica4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511403 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511403 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511405 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511405 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511407 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2511407 INFO  (qtp620300327-18545) [n:127.0.0.1:61499_solr c:implicitcoll s:a  x:implicitcoll_a_replica5] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard a: total=5 found=1 timeoutin=9999ms
   [junit4]   2> 2511407 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2511408 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2511408 INFO  (qtp620300327-18546) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559428078251278336
   [junit4]   2> 2511408 INFO  (searcherExecutor-8839-thread-1-processing-n:127.0.0.1:61499_solr x:implicitcoll_b_replica4 s:b c:implicitcoll) [n:127.0.0.1:61499_solr c:implicitcoll s:b  x:implicitcoll_b_replica4] o.a.s.c.SolrCore [implicitcoll_b_replica4] Registered new searcher Searcher@4fa06f8e[implicitcoll_b_replica4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2511411 INFO  (qtp2094182752-18562) [n:127.0.0.1:61503_solr c:implicitcoll s:a  x:implicitcoll_a_replica4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511411 INFO  (qtp2094182752-18562) [n:127.0.0.1:61503_solr c:implicitcoll s:a  x:implicitcoll_a_replica4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511413 INFO  (qtp697516828-18555) [n:127.0.0.1:61500_solr c:implicitcoll s:b  x:implicitcoll_b_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511413 INFO  (qtp697516828-18555) [n:127.0.0.1:61500_solr c:implicitcoll s:b  x:implicitcoll_b_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511413 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.s.SolrIndexSearcher Opening [Searcher@520e5cb6[implicitcoll_c_replica3] main]
   [junit4]   2> 2511414 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2511414 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2511414 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2511414 INFO  (searcherExecutor-8836-thread-1-processing-n:127.0.0.1:61502_solr x:implicitcoll_a_replica2 s:a c:implicitcoll) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.c.SolrCore [implicitcoll_a_replica2] Registered new searcher Searcher@406fb8b[implicitcoll_a_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2511415 INFO  (qtp124828377-18573) [n:127.0.0.1:61502_solr c:implicitcoll s:a  x:implicitcoll_a_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559428078258618368
   [junit4]   2> 2511415 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2511415 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2511416 INFO  (searcherExecutor-8838-thread-1-processing-n:127.0.0.1:61502_solr x:implicitcoll_c_replica4 s:c c:implicitcoll) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.c.SolrCore [implicitcoll_c_replica4] Registered new searcher Searcher@6d69f2b[implicitcoll_c_replica4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2511416 INFO  (qtp124828377-18575) [n:127.0.0.1:61502_solr c:implicitcoll s:c  x:implicitcoll_c_replica4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559428078259666944
   [junit4]   2> 2511419 INFO  (qtp124828377-18571) [n:127.0.0.1:61502_solr c:implicitcoll s:b  x:implicitcoll_b_replica1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard b: total=5 found=2 timeoutin=9999ms
   [junit4]   2> 2511423 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.s.SolrIndexSearcher Opening [Searcher@36c8a5a9[implicitcoll_b_replica3] main]
   [junit4]   2> 2511424 INFO  (qtp697516828-18555) [n:127.0.0.1:61500_solr c:implicitcoll s:b  x:implicitcoll_b_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2511424 INFO  (qtp697516828-18555) [n:127.0.0.1:61500_solr c:implicitcoll s:b  x:implicitcoll_b_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2511424 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2511425 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2511425 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2511425 INFO  (qtp2094182752-18561) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559428078269104128
   [junit4]   2> 2511425 INFO  (searcherExecutor-8843-thread-1-processing-n:127.0.0.1:61503_solr x:implicitcoll_b_replica3 s:b c:implicitcoll) [n:127.0.0.1:61503_solr c:implicitcoll s:b  x:implicitcoll_b_replica3] o.a.s.c.SolrCore [implicitcoll_b_replica3] Registered new searcher Searcher@36c8a5a9[implicitcoll_b_replica3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2511427 INFO  (qtp697516828-18552) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511427 INFO  (qtp697516828-18552) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511428 INFO  (qtp697516828-18551) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2511428 INFO  (qtp697516828-18551) [n:127.0.0.1:61500_solr c:implicitcoll s:c  x:implicitcoll_c_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2511430 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2511430 INFO  (qtp620300327-18542) [n:127.0.0.1:61499_solr c:implicitcoll s:c  x:implicitcoll_c_replica3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4

[...truncated too long message...]

> 7 /solr/overseer/collection-queue-work
   [junit4]   2> 2 /solr/overseer/queue
   [junit4]   2> 2 /solr/overseer/queue-work
   [junit4]   2>
   [junit4]   2> 3108648 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[723A1655EA4FAD35]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 3108648 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[723A1655EA4FAD35]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for all SolrIndexSearchers to be released
   [junit4]   2> 3108649 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[723A1655EA4FAD35]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=8352, maxDocsPerChunk=825, blockSize=5), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=8352, blockSize=5)), sim=RandomSimilarity(queryNorm=true): {}, locale=ro, timezone=America/Grenada
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=67583928,total=529006592
   [junit4]   2> NOTE: All tests run in this JVM: [FullHLLTest, BigEndianAscendingWordDeserializerTest, CoreMergeIndexesAdminHandlerTest, TestSolrDeletionPolicy2, LeaderFailoverAfterPartitionTest, TestImplicitCoreProperties, TestLuceneMatchVersion, FullSolrCloudDistribCmdsTest, BufferStoreTest, BadCopyFieldTest, TestHdfsBackupRestoreCore, ClusterStateTest, DistributedIntervalFacetingTest, LukeRequestHandlerTest, ShowFileRequestHandlerTest, SuggesterWFSTTest, TestSSLRandomization, TestReversedWildcardFilterFactory, TestSolr4Spatial, ClassificationUpdateProcessorTest, LoggingHandlerTest, DeleteReplicaTest, DistribDocExpirationUpdateProcessorTest, TestCrossCoreJoin, IndexSchemaTest, TestSolrQueryParser, TestSmileRequest, TestPivotHelperCode, BasicDistributedZk2Test, TestRandomFaceting, FieldMutatingUpdateProcessorTest, TestScoreJoinQPScore, TestFieldCollectionResource, TestStressInPlaceUpdates, ReturnFieldsTest, SpellCheckComponentTest, CSVRequestHandlerTest, TestConfigSetsAPIZkFailure, SolrCmdDistributorTest, HighlighterConfigTest, UpdateRequestProcessorFactoryTest, TestCloudSchemaless, TestOrdValues, VMParamsZkACLAndCredentialsProvidersTest, TestNoOpRegenerator, TestConfig, ZkStateWriterTest, SolrJmxReporterTest, DefaultValueUpdateProcessorTest, SolrGangliaReporterTest, HttpSolrCallGetCoreTest, ParsingFieldUpdateProcessorsTest, TestPathTrie, TestRawResponseWriter, BadIndexSchemaTest, TestSimpleTrackingShardHandler, TestDynamicFieldCollectionResource, ShufflingReplicaListTransformerTest, TestFieldCacheWithThreads, TestInPlaceUpdatesDistrib, TestCodecSupport, DistributedQueueTest, TestExactStatsCache, TestCustomStream, AnalysisErrorHandlingTest, TestDownShardTolerantSearch, JavabinLoaderTest, HdfsNNFailoverTest, DocValuesMissingTest, TestClassicSimilarityFactory, RemoteQueryErrorTest, DistributedTermsComponentTest, TestUtils, TestCopyFieldCollectionResource, DistributedFacetPivotLongTailTest, TestSolrCoreSnapshots, JsonValidatorTest, SmileWriterTest, TestSolrCoreProperties, TestDocSet, TestSearchPerf, HdfsTlogReplayBufferedWhileIndexingTest, SpellingQueryConverterTest, UpdateLogTest, CloudMLTQParserTest, HLLUtilTest, HdfsDirectoryFactoryTest, TestWriterPerf, TestGeoJSONResponseWriter, ChaosMonkeySafeLeaderTest, DocExpirationUpdateProcessorFactoryTest, TestAuthenticationFramework, CursorMarkTest, TestFastWriter, CollectionsAPIDistributedZkTest, SolrCoreMetricManagerTest, DocValuesNotIndexedTest, HdfsSyncSliceTest, AddSchemaFieldsUpdateProcessorFactoryTest, HdfsChaosMonkeySafeLeaderTest, TestManagedSchema, ConvertedLegacyTest, TestBulkSchemaAPI, SortByFunctionTest, BJQParserTest, TestPKIAuthenticationPlugin, LeaderInitiatedRecoveryOnCommitTest, HdfsRecoveryZkTest, TestBadConfig, TestQueryUtils, SimpleCollectionCreateDeleteTest, EchoParamsTest, RequestHandlersTest, TestIndexSearcher, TestAddFieldRealTimeGet, TestRequestForwarding, TestSerializedLuceneMatchVersion, TestCustomDocTransformer, TestSolrXml, TestDistribIDF, TestNamedUpdateProcessors, TestFiltering, TestLeaderElectionZkExpiry, ShardRoutingCustomTest, ScriptEngineTest, TestSQLHandler, TestValueSourceCache, TestSchemaSimilarityResource, RegexBoostProcessorTest, DistanceFunctionTest, TestShortCircuitedRequests, UUIDFieldTest, TestCloudManagedSchema, MinimalSchemaTest, TestSchemaNameResource, TestSha256AuthenticationProvider, TestReqParamsAPI, NotRequiredUniqueKeyTest, TestTrieFacet, TestBlendedInfixSuggestions, ExternalFileFieldSortTest, TestTrie, ResourceLoaderTest, TestFuzzyAnalyzedSuggestions, CoreAdminHandlerTest, SolrIndexSplitterTest, ResponseLogComponentTest, ReplicationFactorTest, SuggesterTest, URLClassifyProcessorTest, TestConfigSets, TestManagedResourceStorage, BasicAuthIntegrationTest, CollectionStateFormat2Test, TestImpersonationWithHadoopAuth, CachingDirectoryFactoryTest, BasicFunctionalityTest, TestJettySolrRunner, HttpPartitionTest, TestUnifiedSolrHighlighter, CurrencyFieldOpenExchangeTest, TestLocalFSCloudBackupRestore, OverriddenZkACLAndCredentialsProvidersTest, BlockJoinFacetRandomTest, TestReload, SolrMetricReporterTest, TestReloadAndDeleteDocs, TestStressRecovery, WordBreakSolrSpellCheckerTest, TestNonDefinedSimilarityFactory, RecoveryZkTest, ClusterStateUpdateTest, ZkSolrClientTest, TestRandomDVFaceting, ZkCLITest, AutoCommitTest, TestDistributedGrouping, TestHashPartitioner, TermVectorComponentDistributedTest, TestStressReorder, TestJoin, SimpleFacetsTest, SolrCoreTest, TestGroupingSearch, PeerSyncTest, HighlighterTest, SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SpellCheckCollatorTest, SolrRequestParserTest, SuggesterTSTTest, SolrCoreCheckLockOnStartupTest, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, DocValuesMultiTest, XsltUpdateRequestHandlerTest, DebugComponentTest, DisMaxRequestHandlerTest, DocumentAnalysisRequestHandlerTest, TestOmitPositions, XmlUpdateRequestHandlerTest, TermVectorComponentTest, RequiredFieldsTest, FastVectorHighlighterTest, TestCollationField, TestCSVResponseWriter, QueryParsingTest, JsonLoaderTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, TestLFUCache, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, RAMDirectoryFactoryTest, TestDocumentBuilder, SliceStateTest, CircularListTest, DistributedMLTComponentTest, TestRTGBase, SolrTestCaseJ4Test, TestDistributedMissingSort, TestHighlightDedupGrouping, ConnectionReuseTest, ActionThrottleTest, AssignTest, BaseCdcrDistributedZkTest, CdcrBootstrapTest, CollectionsAPIAsyncDistributedZkTest, DeleteInactiveReplicaTest, DeleteLastCustomShardedReplicaTest, DeleteNodeTest, DeleteShardTest, DeleteStatusTest, LeaderElectionContextKeyTest, LeaderFailureAfterFreshStartTest, MissingSegmentRecoveryTest, OverseerModifyCollectionTest, OverseerStatusTest, PeerSyncReplicationTest, TestLeaderInitiatedRecoveryThread, TestMiniSolrCloudClusterSSL, TestOnReconnectListenerSupport, TestRandomFlRTGCloud, TestRebalanceLeaders, TestSegmentSorting, TestSolrCloudWithDelegationTokens, TestSolrCloudWithSecureImpersonation, TestStressLiveNodes, TestTolerantUpdateProcessorRandomCloud]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestTolerantUpdateProcessorRandomCloud -Dtests.seed=723A1655EA4FAD35 -Dtests.slow=true -Dtests.locale=ro -Dtests.timezone=America/Grenada -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestTolerantUpdateProcessorRandomCloud (suite) <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:64036/solr: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]    > <head>
   [junit4]    > <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
   [junit4]    > <title>Error 404 </title>
   [junit4]    > </head>
   [junit4]    > <body>
   [junit4]    > <h2>HTTP ERROR: 404</h2>
   [junit4]    > <p>Problem accessing /solr/admin/collections. Reason:
   [junit4]    > <pre>    Can not find: /solr/admin/collections</pre></p>
   [junit4]    > <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/>
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > at __randomizedtesting.SeedInfo.seed([723A1655EA4FAD35]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:439)
   [junit4]    > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:391)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1358)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1109)
   [junit4]    > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
   [junit4]    > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
   [junit4]    > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
   [junit4]    > at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorRandomCloud.java:111)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [532/694 (4!)] on J1 in 10.10s, 0 tests, 1 error <<< FAILURES!

[...truncated 63093 lines...]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3836 - Failure!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3836/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

3 tests failed.
FAILED:  org.apache.solr.handler.TestReqParamsAPI.test

Error Message:
Could not get expected value  'CY val' for path 'response/params/y/c' full output: {   "responseHeader":{     "status":0,     "QTime":0},   "response":{"znodeVersion":-1}},  from server:  http://127.0.0.1:60200/solr/collection1_shard1_replica2

Stack Trace:
java.lang.AssertionError: Could not get expected value  'CY val' for path 'response/params/y/c' full output: {
  "responseHeader":{
    "status":0,
    "QTime":0},
  "response":{"znodeVersion":-1}},  from server:  http://127.0.0.1:60200/solr/collection1_shard1_replica2
        at __randomizedtesting.SeedInfo.seed([A33AFB76746001BE:2B6EC4ACDA9C6C46]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:556)
        at org.apache.solr.handler.TestReqParamsAPI.testReqParams(TestReqParamsAPI.java:174)
        at org.apache.solr.handler.TestReqParamsAPI.test(TestReqParamsAPI.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.search.TestRecovery.testCorruptLog

Error Message:
mismatch: '3'!='0' @ response/numFound

Stack Trace:
java.lang.RuntimeException: mismatch: '3'!='0' @ response/numFound
        at __randomizedtesting.SeedInfo.seed([A33AFB76746001BE:51E74FA58E585062]:0)
        at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:1006)
        at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:953)
        at org.apache.solr.search.TestRecovery.testCorruptLog(TestRecovery.java:1274)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter

Error Message:
Collection not found: withShardField

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: withShardField
        at __randomizedtesting.SeedInfo.seed([A33AFB76746001BE:F66A13E4D899CE4E]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1376)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1072)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:232)
        at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11349 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReqParamsAPI
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/init-core-data-001
   [junit4]   2> 779909 INFO  (SUITE-TestReqParamsAPI-seed#[A33AFB76746001BE]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 779912 INFO  (SUITE-TestReqParamsAPI-seed#[A33AFB76746001BE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 779913 INFO  (SUITE-TestReqParamsAPI-seed#[A33AFB76746001BE]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001
   [junit4]   2> 779913 INFO  (SUITE-TestReqParamsAPI-seed#[A33AFB76746001BE]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 779914 INFO  (Thread-1176) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 779914 INFO  (Thread-1176) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 780016 INFO  (SUITE-TestReqParamsAPI-seed#[A33AFB76746001BE]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:60197
   [junit4]   2> 780039 INFO  (jetty-launcher-907-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 780041 INFO  (jetty-launcher-907-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 780044 INFO  (jetty-launcher-907-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@31550ad0{/solr,null,AVAILABLE}
   [junit4]   2> 780044 INFO  (jetty-launcher-907-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7c64646f{/solr,null,AVAILABLE}
   [junit4]   2> 780045 INFO  (jetty-launcher-907-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@457e5cba{HTTP/1.1,[http/1.1]}{127.0.0.1:60199}
   [junit4]   2> 780046 INFO  (jetty-launcher-907-thread-2) [    ] o.e.j.s.Server Started @785316ms
   [junit4]   2> 780046 INFO  (jetty-launcher-907-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60199}
   [junit4]   2> 780046 INFO  (jetty-launcher-907-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3d87c2a1{HTTP/1.1,[http/1.1]}{127.0.0.1:60200}
   [junit4]   2> 780046 INFO  (jetty-launcher-907-thread-1) [    ] o.e.j.s.Server Started @785316ms
   [junit4]   2> 780046 ERROR (jetty-launcher-907-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 780046 INFO  (jetty-launcher-907-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60200}
   [junit4]   2> 780046 ERROR (jetty-launcher-907-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 780047 INFO  (jetty-launcher-907-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 780047 INFO  (jetty-launcher-907-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 780047 INFO  (jetty-launcher-907-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 780047 INFO  (jetty-launcher-907-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-16T04:56:58.514Z
   [junit4]   2> 780048 INFO  (jetty-launcher-907-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 780048 INFO  (jetty-launcher-907-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 780048 INFO  (jetty-launcher-907-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 780048 INFO  (jetty-launcher-907-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-16T04:56:58.515Z
   [junit4]   2> 780053 INFO  (jetty-launcher-907-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 780053 INFO  (jetty-launcher-907-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 780070 INFO  (jetty-launcher-907-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60197/solr
   [junit4]   2> 780074 INFO  (jetty-launcher-907-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60197/solr
   [junit4]   2> 780119 INFO  (jetty-launcher-907-thread-2) [n:127.0.0.1:60199_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 780119 INFO  (jetty-launcher-907-thread-1) [n:127.0.0.1:60200_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 780122 INFO  (jetty-launcher-907-thread-2) [n:127.0.0.1:60199_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60199_solr
   [junit4]   2> 780123 INFO  (jetty-launcher-907-thread-1) [n:127.0.0.1:60200_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60200_solr
   [junit4]   2> 780124 INFO  (jetty-launcher-907-thread-2) [n:127.0.0.1:60199_solr    ] o.a.s.c.Overseer Overseer (id=97466516660748293-127.0.0.1:60199_solr-n_0000000000) starting
   [junit4]   2> 780129 INFO  (zkCallback-917-thread-1-processing-n:127.0.0.1:60199_solr) [n:127.0.0.1:60199_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 780129 INFO  (zkCallback-918-thread-1-processing-n:127.0.0.1:60200_solr) [n:127.0.0.1:60200_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 780154 INFO  (jetty-launcher-907-thread-2) [n:127.0.0.1:60199_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60199_solr
   [junit4]   2> 780163 INFO  (zkCallback-918-thread-1-processing-n:127.0.0.1:60200_solr) [n:127.0.0.1:60200_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 780163 INFO  (zkCallback-917-thread-1-processing-n:127.0.0.1:60199_solr) [n:127.0.0.1:60199_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 780253 INFO  (jetty-launcher-907-thread-2) [n:127.0.0.1:60199_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node2/.
   [junit4]   2> 780253 INFO  (jetty-launcher-907-thread-1) [n:127.0.0.1:60200_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/.
   [junit4]   2> 780618 INFO  (SUITE-TestReqParamsAPI-seed#[A33AFB76746001BE]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 780621 INFO  (SUITE-TestReqParamsAPI-seed#[A33AFB76746001BE]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60197/solr ready
   [junit4]   2> 780634 INFO  (qtp1625587029-5612) [n:127.0.0.1:60199_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf1&name=collection1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 780643 INFO  (OverseerThreadFactory-2373-thread-1-processing-n:127.0.0.1:60199_solr) [n:127.0.0.1:60199_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 780761 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard1_replica2&action=CREATE&numShards=1&collection=collection1&shard=shard1&wt=javabin&version=2
   [junit4]   2> 780761 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard1_replica1&action=CREATE&numShards=1&collection=collection1&shard=shard1&wt=javabin&version=2
   [junit4]   2> 780874 INFO  (zkCallback-917-thread-2-processing-n:127.0.0.1:60199_solr) [n:127.0.0.1:60199_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 780874 INFO  (zkCallback-918-thread-1-processing-n:127.0.0.1:60200_solr) [n:127.0.0.1:60200_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 781780 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 781780 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 781787 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=minimal
   [junit4]   2> 781787 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=minimal
   [junit4]   2> 781792 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 781792 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 781794 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
   [junit4]   2> 781794 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica1' using configuration from collection collection1
   [junit4]   2> 781795 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/collection1_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/./collection1_shard1_replica2/data/]
   [junit4]   2> 781795 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.SolrCore [[collection1_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node2/collection1_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node2/./collection1_shard1_replica1/data/]
   [junit4]   2> 781878 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 781878 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 781878 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 781878 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 781880 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 781880 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 781880 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 781880 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 781880 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@76270b2e[collection1_shard1_replica2] main]
   [junit4]   2> 781880 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7f51ea62[collection1_shard1_replica1] main]
   [junit4]   2> 781882 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 781882 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 781883 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 781883 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 781883 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 781883 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 781884 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 781884 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 781884 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 781884 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 781884 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559464268635570176
   [junit4]   2> 781884 INFO  (searcherExecutor-2378-thread-1-processing-n:127.0.0.1:60200_solr x:collection1_shard1_replica2 s:shard1 c:collection1) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@76270b2e[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 781884 INFO  (searcherExecutor-2379-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@7f51ea62[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 781884 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559464268635570176
   [junit4]   2> 781895 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 781895 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 781895 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60200/solr/collection1_shard1_replica2/
   [junit4]   2> 781895 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica2 url=http://127.0.0.1:60200/solr START replicas=[http://127.0.0.1:60199/solr/collection1_shard1_replica1/] nUpdates=100
   [junit4]   2> 781896 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica2 url=http://127.0.0.1:60200/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 781898 INFO  (qtp1625587029-5616) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 781899 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 781899 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 781906 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60200/solr/collection1_shard1_replica2/ shard1
   [junit4]   2> 782013 INFO  (zkCallback-917-thread-1-processing-n:127.0.0.1:60199_solr) [n:127.0.0.1:60199_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 782013 INFO  (zkCallback-918-thread-1-processing-n:127.0.0.1:60200_solr) [n:127.0.0.1:60200_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 782015 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 782020 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard1_replica2&action=CREATE&numShards=1&collection=collection1&shard=shard1&wt=javabin&version=2} status=0 QTime=1259
   [junit4]   2> 782134 INFO  (zkCallback-917-thread-3-processing-n:127.0.0.1:60199_solr) [n:127.0.0.1:60199_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 782134 INFO  (zkCallback-918-thread-1-processing-n:127.0.0.1:60200_solr) [n:127.0.0.1:60200_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 782900 INFO  (qtp1625587029-5614) [n:127.0.0.1:60199_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard1_replica1&action=CREATE&numShards=1&collection=collection1&shard=shard1&wt=javabin&version=2} status=0 QTime=2139
   [junit4]   2> 782906 INFO  (qtp1625587029-5612) [n:127.0.0.1:60199_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 783004 INFO  (zkCallback-918-thread-1-processing-n:127.0.0.1:60200_solr) [n:127.0.0.1:60200_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 783004 INFO  (zkCallback-917-thread-2-processing-n:127.0.0.1:60199_solr) [n:127.0.0.1:60199_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 783908 INFO  (qtp1625587029-5612) [n:127.0.0.1:60199_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf1&name=collection1&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=3274
   [junit4]   2> 783930 INFO  (TEST-TestReqParamsAPI.test-seed#[A33AFB76746001BE]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 783944 INFO  (TEST-TestReqParamsAPI.test-seed#[A33AFB76746001BE]) [    ] o.a.s.c.TestSolrConfigHandler going to send config command. path /config?wt=json , payload: {
   [junit4]   2> 'create-requesthandler' : { 'name' : '/dump0', 'class': 'org.apache.solr.handler.DumpRequestHandler' }
   [junit4]   2> }
   [junit4]   2> 783952 INFO  (qtp1625587029-5608) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Executed config commands successfully and persisted to ZK [{"create-requesthandler":{
   [junit4]   2>     "name":"/dump0",
   [junit4]   2>     "class":"org.apache.solr.handler.DumpRequestHandler"}}]
   [junit4]   2> 783953 INFO  (Thread-1195) [n:127.0.0.1:60199_solr    ] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica1
   [junit4]   2> 783953 INFO  (Thread-1196) [n:127.0.0.1:60200_solr    ] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica2
   [junit4]   2> 783955 INFO  (Thread-1195) [n:127.0.0.1:60199_solr    ] o.a.s.c.SolrCore core reload collection1_shard1_replica1
   [junit4]   2> 783955 INFO  (Thread-1196) [n:127.0.0.1:60200_solr    ] o.a.s.c.SolrCore core reload collection1_shard1_replica2
   [junit4]   2> 783957 INFO  (qtp1625587029-5608) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Waiting up to 30 secs for 2 replicas to set the property overlay to be of version 0 for collection collection1
   [junit4]   2> 783960 INFO  (solrHandlerExecutor-2388-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 783961 INFO  (solrHandlerExecutor-2388-thread-2-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 783969 INFO  (Thread-1196) [n:127.0.0.1:60200_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 783969 INFO  (Thread-1195) [n:127.0.0.1:60199_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 783978 INFO  (Thread-1196) [n:127.0.0.1:60200_solr    ] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=minimal
   [junit4]   2> 783978 INFO  (Thread-1195) [n:127.0.0.1:60199_solr    ] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=minimal
   [junit4]   2> 783982 INFO  (Thread-1196) [n:127.0.0.1:60200_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 783982 INFO  (Thread-1195) [n:127.0.0.1:60199_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 783985 INFO  (Thread-1195) [n:127.0.0.1:60199_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica1' using configuration from collection collection1
   [junit4]   2> 783985 INFO  (Thread-1196) [n:127.0.0.1:60200_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
   [junit4]   2> 783987 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/collection1_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/./collection1_shard1_replica2/data/]
   [junit4]   2> 783987 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [[collection1_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node2/collection1_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node2/./collection1_shard1_replica1/data/]
   [junit4]   2> 784065 INFO  (qtp1625587029-5613) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 784065 INFO  (qtp110594021-5606) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 784066 INFO  (qtp1625587029-5613) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 784066 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Trying to update my configs
   [junit4]   2> 784067 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica1
   [junit4]   2> 784067 INFO  (qtp110594021-5606) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.S.Request [collection1_shard1_replica2]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 784067 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler Trying to update my configs
   [junit4]   2> 784067 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica2
   [junit4]   2> 784068 INFO  (solrHandlerExecutor-2388-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:60199/solr/collection1_shard1_replica1/ for prop overlay   after 1 attempts
   [junit4]   2> 784068 INFO  (solrHandlerExecutor-2388-thread-2-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:60200/solr/collection1_shard1_replica2/ for prop overlay   after 1 attempts
   [junit4]   2> 784068 INFO  (solrHandlerExecutor-2388-thread-2-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 784068 INFO  (solrHandlerExecutor-2388-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 784069 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore core reload collection1_shard1_replica1
   [junit4]   2> 784069 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore core reload collection1_shard1_replica2
   [junit4]   2> 784079 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 784079 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 784079 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2cd5efa7[collection1_shard1_replica1] main]
   [junit4]   2> 784079 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 784079 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 784080 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@655a1b4b[collection1_shard1_replica2] main]
   [junit4]   2> 784081 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 784082 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 784082 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 784083 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 784083 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 784083 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 784084 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 784084 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 784084 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559464270942437376
   [junit4]   2> 784084 INFO  (searcherExecutor-2390-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@2cd5efa7[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784085 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 784085 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 784085 INFO  (searcherExecutor-2389-thread-1-processing-n:127.0.0.1:60200_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@655a1b4b[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784086 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559464270944534528
   [junit4]   2> 784089 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 784090 INFO  (Thread-1195) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6e6a6a2a[collection1_shard1_replica1] main]
   [junit4]   2> 784090 INFO  (searcherExecutor-2390-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@6e6a6a2a[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784091 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 784091 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@630ec08d[collection1_shard1_replica2] main]
   [junit4]   2> 784092 INFO  (searcherExecutor-2389-thread-1-processing-n:127.0.0.1:60200_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@630ec08d[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784092 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2]  CLOSING SolrCore org.apache.solr.core.SolrCore@3c68900
   [junit4]   2> 784094 INFO  (Thread-1196) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica2
   [junit4]   2> 784105 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 784106 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 784112 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=minimal
   [junit4]   2> 784114 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=minimal
   [junit4]   2> 784116 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 784117 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 784118 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica1' using configuration from collection collection1
   [junit4]   2> 784118 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
   [junit4]   2> 784119 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [[collection1_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node2/collection1_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node2/./collection1_shard1_replica1/data/]
   [junit4]   2> 784119 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/collection1_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/./collection1_shard1_replica2/data/]
   [junit4]   2> 784175 INFO  (qtp1625587029-5615) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler I already have the expected version 0 of config
   [junit4]   2> 784175 INFO  (qtp1625587029-5615) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
   [junit4]   2> 784175 INFO  (qtp1625587029-5615) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 784175 INFO  (qtp110594021-5607) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler I already have the expected version 0 of config
   [junit4]   2> 784176 INFO  (qtp110594021-5607) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
   [junit4]   2> 784176 INFO  (qtp110594021-5607) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.S.Request [collection1_shard1_replica2]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 784177 INFO  (qtp1625587029-5608) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Took 225.0ms to set the property overlay to be of version 0 for collection collection1
   [junit4]   2> 784177 INFO  (qtp1625587029-5608) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config params={wt=json} status=0 QTime=231
   [junit4]   2> 784178 INFO  (qtp1625587029-5608) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6f6af93f
   [junit4]   2> 784178 INFO  (qtp1625587029-5608) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica1
   [junit4]   2> 784179 INFO  (TEST-TestReqParamsAPI.test-seed#[A33AFB76746001BE]) [    ] o.a.s.c.TestSolrConfigHandler going to send config command. path /config?wt=json , payload: {
   [junit4]   2> 'create-requesthandler' : { 'name' : '/dump1', 'class': 'org.apache.solr.handler.DumpRequestHandler', 'useParams':'x' }
   [junit4]   2> }
   [junit4]   2> 784181 INFO  (qtp1625587029-5616) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.ZkController Persisted config data to node /configs/conf1/configoverlay.json
   [junit4]   2> 784182 INFO  (qtp1625587029-5616) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Executed config commands successfully and persisted to ZK [{"create-requesthandler":{
   [junit4]   2>     "name":"/dump1",
   [junit4]   2>     "class":"org.apache.solr.handler.DumpRequestHandler",
   [junit4]   2>     "useParams":"x"}}]
   [junit4]   2> 784182 INFO  (qtp1625587029-5616) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Waiting up to 30 secs for 2 replicas to set the property overlay to be of version 1 for collection collection1
   [junit4]   2> 784184 INFO  (Thread-1198) [n:127.0.0.1:60200_solr    ] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica2
   [junit4]   2> 784184 INFO  (Thread-1197) [n:127.0.0.1:60199_solr    ] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica1
   [junit4]   2> 784186 INFO  (Thread-1198) [n:127.0.0.1:60200_solr    ] o.a.s.c.SolrCore core reload collection1_shard1_replica2
   [junit4]   2> 784186 INFO  (Thread-1197) [n:127.0.0.1:60199_solr    ] o.a.s.c.SolrCore core reload collection1_shard1_replica1
   [junit4]   2> 784187 INFO  (solrHandlerExecutor-2399-thread-2-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 784187 INFO  (solrHandlerExecutor-2399-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 784224 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 784224 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 784225 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@3fa6e45a[collection1_shard1_replica2] main]
   [junit4]   2> 784226 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 784226 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 784226 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 784226 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7d8fcff8[collection1_shard1_replica1] main]
   [junit4]   2> 784227 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 784227 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 784228 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 784228 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 784228 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 784229 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559464271094480896
   [junit4]   2> 784229 INFO  (searcherExecutor-2398-thread-1-processing-n:127.0.0.1:60200_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@3fa6e45a[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784230 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 784230 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 784231 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 784231 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 784231 INFO  (searcherExecutor-2397-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@7d8fcff8[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784231 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559464271096578048
   [junit4]   2> 784234 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 784235 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@34da02f7[collection1_shard1_replica2] main]
   [junit4]   2> 784235 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2]  CLOSING SolrCore org.apache.solr.core.SolrCore@7264691b
   [junit4]   2> 784235 INFO  (searcherExecutor-2398-thread-1-processing-n:127.0.0.1:60200_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@34da02f7[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784235 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 784235 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica2
   [junit4]   2> 784237 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3c8b2fe[collection1_shard1_replica1] main]
   [junit4]   2> 784237 INFO  (searcherExecutor-2397-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@3c8b2fe[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784254 INFO  (Thread-1198) [n:127.0.0.1:60200_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 784255 INFO  (Thread-1197) [n:127.0.0.1:60199_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 784267 INFO  (Thread-1197) [n:127.0.0.1:60199_solr    ] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=minimal
   [junit4]   2> 784269 INFO  (Thread-1198) [n:127.0.0.1:60200_solr    ] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=minimal
   [junit4]   2> 784272 INFO  (Thread-1198) [n:127.0.0.1:60200_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 784273 INFO  (Thread-1197) [n:127.0.0.1:60199_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 784280 INFO  (Thread-1198) [n:127.0.0.1:60200_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
   [junit4]   2> 784280 INFO  (Thread-1197) [n:127.0.0.1:60199_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica1' using configuration from collection collection1
   [junit4]   2> 784294 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/collection1_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/./collection1_shard1_replica2/data/]
   [junit4]   2> 784295 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [[collection1_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node2/collection1_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node2/./collection1_shard1_replica1/data/]
   [junit4]   2> 784299 INFO  (qtp1625587029-5617) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler expecting overlay version 1 but my version is 0
   [junit4]   2> 784299 INFO  (qtp1625587029-5617) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 784300 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Trying to update my configs
   [junit4]   2> 784300 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica1
   [junit4]   2> 784300 INFO  (solrHandlerExecutor-2399-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 1 from http://127.0.0.1:60199/solr/collection1_shard1_replica1/ for prop overlay   after 1 attempts
   [junit4]   2> 784300 INFO  (solrHandlerExecutor-2399-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 784301 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler expecting overlay version 1 but my version is 0
   [junit4]   2> 784301 INFO  (qtp110594021-5604) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.S.Request [collection1_shard1_replica2]  webapp=/solr path=/config/znodeVersion params={overlay=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 784301 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler Trying to update my configs
   [junit4]   2> 784301 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica2
   [junit4]   2> 784301 INFO  (solrHandlerExecutor-2399-thread-2-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 1 from http://127.0.0.1:60200/solr/collection1_shard1_replica2/ for prop overlay   after 1 attempts
   [junit4]   2> 784301 INFO  (solrHandlerExecutor-2399-thread-2-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 784302 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore core reload collection1_shard1_replica1
   [junit4]   2> 784302 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore core reload collection1_shard1_replica2
   [junit4]   2> 784410 INFO  (qtp110594021-5605) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler expecting overlay version 1 but my version is 0
   [junit4]   2> 784410 INFO  (qtp110594021-5605) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.S.Request [collection1_shard1_replica2]  webapp=/solr path=/config/znodeVersion params={overlay=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 784411 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 784411 INFO  (solrHandlerExecutor-2399-thread-2-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 1 from http://127.0.0.1:60200/solr/collection1_shard1_replica2/ for prop overlay   after 2 attempts
   [junit4]   2> 784411 INFO  (solrHandlerExecutor-2399-thread-2-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 784421 INFO  (qtp1625587029-5613) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler expecting overlay version 1 but my version is 0
   [junit4]   2> 784422 INFO  (qtp1625587029-5613) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 784422 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 784422 INFO  (solrHandlerExecutor-2399-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 1 from http://127.0.0.1:60199/solr/collection1_shard1_replica1/ for prop overlay   after 2 attempts
   [junit4]   2> 784422 INFO  (solrHandlerExecutor-2399-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 784516 INFO  (qtp110594021-5606) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler expecting overlay version 1 but my version is 0
   [junit4]   2> 784516 INFO  (qtp110594021-5606) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.S.Request [collection1_shard1_replica2]  webapp=/solr path=/config/znodeVersion params={overlay=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 784516 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 784516 INFO  (solrHandlerExecutor-2399-thread-2-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 1 from http://127.0.0.1:60200/solr/collection1_shard1_replica2/ for prop overlay   after 3 attempts
   [junit4]   2> 784517 INFO  (solrHandlerExecutor-2399-thread-2-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 784520 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 784520 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 784520 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@7ae5fd90[collection1_shard1_replica2] main]
   [junit4]   2> 784523 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 784524 INFO  (qtp1625587029-5615) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler expecting overlay version 1 but my version is 0
   [junit4]   2> 784524 INFO  (qtp1625587029-5615) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 784524 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 784525 INFO  (solrHandlerExecutor-2399-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 1 from http://127.0.0.1:60199/solr/collection1_shard1_replica1/ for prop overlay   after 3 attempts
   [junit4]   2> 784525 INFO  (solrHandlerExecutor-2399-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 784525 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 784525 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 784529 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 784529 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 784530 INFO  (searcherExecutor-2406-thread-1-processing-n:127.0.0.1:60200_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@7ae5fd90[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784530 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559464271410102272
   [junit4]   2> 784560 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 784560 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 784560 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@12e67d11[collection1_shard1_replica1] main]
   [junit4]   2> 784563 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 784564 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 784564 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 784565 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 784566 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 784566 INFO  (searcherExecutor-2407-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@12e67d11[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784566 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559464271447851008
   [junit4]   2> 784569 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 784569 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@69f1749e[collection1_shard1_replica2] main]
   [junit4]   2> 784569 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2]  CLOSING SolrCore org.apache.solr.core.SolrCore@16826ca3
   [junit4]   2> 784569 INFO  (searcherExecutor-2406-thread-1-processing-n:127.0.0.1:60200_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@69f1749e[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784570 INFO  (Thread-1198) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica2
   [junit4]   2> 784570 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 784570 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1c4f27c6[collection1_shard1_replica1] main]
   [junit4]   2> 784571 INFO  (searcherExecutor-2407-thread-1-processing-n:127.0.0.1:60199_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@1c4f27c6[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 784571 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@c800d9b
   [junit4]   2> 784571 INFO  (Thread-1197) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica1
   [junit4]   2> 784582 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 784583 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 784588 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=minimal
   [junit4]   2> 784592 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 784593 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=minimal
   [junit4]   2> 784595 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 784597 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
   [junit4]   2> 784598 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60200_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/collection1_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.TestReqParamsAPI_A33AFB76746001BE-001/tempDir-001/node1/./collection1_shard1_replica2/data/]
   [junit4]   2> 784597 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:60199_solr c:collection1 s:shard1 r:core_node

[...truncated too long message...]


 [ecj-lint]         ^^
 [ecj-lint] Resource leak: 'a2' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 10. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/response/TestBinaryResponseWriter.java (at line 63)
 [ecj-lint] NamedList res = (NamedList) new JavaBinCodec().unmarshal(new ByteArrayInputStream(baos.toByteArray()));
 [ecj-lint]                            ^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 11. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/response/TestJavabinTupleStreamParser.java (at line 72)
 [ecj-lint] JavabinTupleStreamParser parser = new JavabinTupleStreamParser(new ByteArrayInputStream(bytes), true);
 [ecj-lint]                         ^^^^^^
 [ecj-lint] Resource leak: 'parser' is never closed
 [ecj-lint] ----------
 [ecj-lint] 12. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/response/TestJavabinTupleStreamParser.java (at line 173)
 [ecj-lint] new JavaBinCodec().marshal(response.getValues(), baos);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] 13. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/response/TestJavabinTupleStreamParser.java (at line 175)
 [ecj-lint] Object o = new JavaBinCodec().unmarshal(new ByteArrayInputStream(bytes));
 [ecj-lint]           ^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] 14. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/response/TestJavabinTupleStreamParser.java (at line 178)
 [ecj-lint] JavabinTupleStreamParser parser = new JavabinTupleStreamParser(new ByteArrayInputStream(bytes), false);
 [ecj-lint]                         ^^^^^^
 [ecj-lint] Resource leak: 'parser' is never closed
 [ecj-lint] ----------
 [ecj-lint] 15. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/response/TestJavabinTupleStreamParser.java (at line 192)
 [ecj-lint] new JavaBinCodec().marshal(response.getValues(), baos);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 16. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/response/TestPushWriter.java (at line 57)
 [ecj-lint] m = (Map) new JavaBinCodec().unmarshal(new ByteArrayInputStream(baos.toByteArray()));
 [ecj-lint]          ^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 17. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/response/transform/TestSubQueryTransformer.java (at line 387)
 [ecj-lint] unmarshalled = (NamedList<Object>) new JavaBinCodec().unmarshal(
 [ecj-lint]                                   ^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 18. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/search/TestDocSet.java (at line 241)
 [ecj-lint] return loadfactor!=0 ? new HashDocSet(a,0,n,1/loadfactor) : new HashDocSet(a,0,n);
 [ecj-lint]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] 19. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/search/TestDocSet.java (at line 526)
 [ecj-lint] DocSet a = new BitDocSet(bs);
 [ecj-lint]       ^
 [ecj-lint] Resource leak: 'a' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 20. ERROR in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/search/TestRecovery.java (at line 26)
 [ecj-lint] import org.apache.solr.common.SolrInputDocument;
 [ecj-lint]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.common.SolrInputDocument is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 21. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/security/TestPKIAuthenticationPlugin.java (at line 85)
 [ecj-lint] final MockPKIAuthenticationPlugin mock = new MockPKIAuthenticationPlugin(null, nodeName);
 [ecj-lint]                                  ^^^^
 [ecj-lint] Resource leak: 'mock' is never closed
 [ecj-lint] ----------
 [ecj-lint] 22. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/security/TestPKIAuthenticationPlugin.java (at line 138)
 [ecj-lint] MockPKIAuthenticationPlugin mock1 = new MockPKIAuthenticationPlugin(null, nodeName) {
 [ecj-lint]                            ^^^^^
 [ecj-lint] Resource leak: 'mock1' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 23. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/security/TestRuleBasedAuthorizationPlugin.java (at line 365)
 [ecj-lint] RuleBasedAuthorizationPlugin plugin = new RuleBasedAuthorizationPlugin();
 [ecj-lint]                             ^^^^^^
 [ecj-lint] Resource leak: 'plugin' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 24. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/security/TestSha256AuthenticationProvider.java (at line 49)
 [ecj-lint] BasicAuthPlugin basicAuthPlugin = new BasicAuthPlugin();
 [ecj-lint]                ^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'basicAuthPlugin' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 25. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/spelling/SimpleQueryConverter.java (at line 42)
 [ecj-lint] WhitespaceAnalyzer analyzer = new WhitespaceAnalyzer();
 [ecj-lint]                   ^^^^^^^^
 [ecj-lint] Resource leak: 'analyzer' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 26. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheVsDocValues.java (at line 139)
 [ecj-lint] IndexWriter w = new IndexWriter(d, newIndexWriterConfig(analyzer));
 [ecj-lint]            ^
 [ecj-lint] Resource leak: 'w' is never closed
 [ecj-lint] ----------
 [ecj-lint] 27. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheVsDocValues.java (at line 172)
 [ecj-lint] throw iae;
 [ecj-lint] ^^^^^^^^^^
 [ecj-lint] Resource leak: 'w' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 28. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheVsDocValues.java (at line 178)
 [ecj-lint] return;
 [ecj-lint] ^^^^^^^
 [ecj-lint] Resource leak: 'w' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 29. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/update/AddBlockUpdateTest.java (at line 415)
 [ecj-lint] new JavaBinCodec().marshal(topDocument, os);
 [ecj-lint] ^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] 30. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/update/AddBlockUpdateTest.java (at line 419)
 [ecj-lint] SolrInputDocument result = (SolrInputDocument) new JavaBinCodec().unmarshal(is);
 [ecj-lint]                                               ^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 31. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/update/SolrIndexSplitterTest.java (at line 181)
 [ecj-lint] EmbeddedSolrServer server1 = new EmbeddedSolrServer(h.getCoreContainer(), "split1");
 [ecj-lint]                   ^^^^^^^
 [ecj-lint] Resource leak: 'server1' is never closed
 [ecj-lint] ----------
 [ecj-lint] 32. WARNING in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/update/SolrIndexSplitterTest.java (at line 182)
 [ecj-lint] EmbeddedSolrServer server2 = new EmbeddedSolrServer(h.getCoreContainer(), "split2");
 [ecj-lint]                   ^^^^^^^
 [ecj-lint] Resource leak: 'server2' is never closed
 [ecj-lint] ----------
 [ecj-lint] 32 problems (1 error, 31 warnings)

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:775: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:101: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build.xml:674: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2010: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2043: Compile failed; see the compiler error output for details.

Total time: 111 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3837 - Still unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3837/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.PeerSyncReplicationTest.test

Error Message:
timeout waiting to see all nodes active

Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
        at __randomizedtesting.SeedInfo.seed([95AAC0766C0D9AED:1DFEFFACC2F1F715]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
        at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
        at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
        at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12391 lines...]
   [junit4] Suite: org.apache.solr.cloud.PeerSyncReplicationTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/init-core-data-001
   [junit4]   2> 2441738 INFO  (SUITE-PeerSyncReplicationTest-seed#[95AAC0766C0D9AED]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 2441742 INFO  (SUITE-PeerSyncReplicationTest-seed#[95AAC0766C0D9AED]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2441742 INFO  (SUITE-PeerSyncReplicationTest-seed#[95AAC0766C0D9AED]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2441744 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2441744 INFO  (Thread-3114) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2441744 INFO  (Thread-3114) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2441850 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.ZkTestServer start zk server on port:54482
   [junit4]   2> 2441884 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2441997 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2442053 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2442066 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2442091 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2442106 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2442116 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2442133 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2442155 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2442180 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2442222 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2446844 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/control-001/cores/collection1
   [junit4]   2> 2446892 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2446894 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7b335a8c{/,null,AVAILABLE}
   [junit4]   2> 2446896 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1e96f181{SSL,[ssl, http/1.1]}{127.0.0.1:54514}
   [junit4]   2> 2446896 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.Server Started @2451821ms
   [junit4]   2> 2446896 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/tempDir-001/control/data, hostContext=/, hostPort=54514, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/control-001/cores}
   [junit4]   2> 2446897 ERROR (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2446897 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2446897 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2446897 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2446897 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-16T15:59:44.274Z
   [junit4]   2> 2446901 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2446902 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/control-001/solr.xml
   [junit4]   2> 2446917 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54482/solr
   [junit4]   2> 2446957 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54514_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2446958 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54514_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54514_
   [junit4]   2> 2446960 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54514_    ] o.a.s.c.Overseer Overseer (id=97469122420670469-127.0.0.1:54514_-n_0000000000) starting
   [junit4]   2> 2446974 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54514_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54514_
   [junit4]   2> 2446980 INFO  (zkCallback-2464-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2447084 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54514_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/control-001/cores
   [junit4]   2> 2447084 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54514_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2447088 INFO  (OverseerStateUpdate-97469122420670469-127.0.0.1:54514_-n_0000000000) [n:127.0.0.1:54514_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2448116 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2448133 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2448200 WARN  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2448202 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2448221 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2448221 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/control-001/cores/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/control-001/cores/collection1/data/]
   [junit4]   2> 2448221 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f152046
   [junit4]   2> 2448224 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=5.0546875, floorSegmentMB=2.1708984375, forceMergeDeletesPctAllowed=17.25462231611045, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2448242 WARN  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2448317 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2448317 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2448318 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2448318 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2448318 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6265900090466819]
   [junit4]   2> 2448319 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@736d370a[collection1] main]
   [junit4]   2> 2448321 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2448321 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2448321 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2448323 INFO  (searcherExecutor-7861-thread-1-processing-n:127.0.0.1:54514_ x:collection1 c:control_collection) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@736d370a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2448324 INFO  (coreLoadExecutor-7860-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559505965627211776
   [junit4]   2> 2448334 INFO  (coreZkRegister-7853-thread-1-processing-n:127.0.0.1:54514_ x:collection1 c:control_collection) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2448335 INFO  (coreZkRegister-7853-thread-1-processing-n:127.0.0.1:54514_ x:collection1 c:control_collection) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2448335 INFO  (coreZkRegister-7853-thread-1-processing-n:127.0.0.1:54514_ x:collection1 c:control_collection) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:54514/collection1/
   [junit4]   2> 2448335 INFO  (coreZkRegister-7853-thread-1-processing-n:127.0.0.1:54514_ x:collection1 c:control_collection) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2448335 INFO  (coreZkRegister-7853-thread-1-processing-n:127.0.0.1:54514_ x:collection1 c:control_collection) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:54514/collection1/ has no replicas
   [junit4]   2> 2448342 INFO  (coreZkRegister-7853-thread-1-processing-n:127.0.0.1:54514_ x:collection1 c:control_collection) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:54514/collection1/ shard1
   [junit4]   2> 2448507 INFO  (coreZkRegister-7853-thread-1-processing-n:127.0.0.1:54514_ x:collection1 c:control_collection) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2448741 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2448743 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54482/solr ready
   [junit4]   2> 2448743 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2448743 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 2449086 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-1-001/cores/collection1
   [junit4]   2> 2449087 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-1-001
   [junit4]   2> 2449092 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2449093 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@58f59991{/,null,AVAILABLE}
   [junit4]   2> 2449094 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4e8bf5e3{SSL,[ssl, http/1.1]}{127.0.0.1:54520}
   [junit4]   2> 2449094 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.Server Started @2454018ms
   [junit4]   2> 2449094 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=54520, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-1-001/cores}
   [junit4]   2> 2449095 ERROR (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2449095 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2449095 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2449095 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2449095 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-16T15:59:46.472Z
   [junit4]   2> 2449099 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2449100 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-1-001/solr.xml
   [junit4]   2> 2449111 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54482/solr
   [junit4]   2> 2449129 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2449134 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54520_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2449138 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54520_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54520_
   [junit4]   2> 2449141 INFO  (zkCallback-2464-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2449142 INFO  (zkCallback-2468-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2449142 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2449273 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54520_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-1-001/cores
   [junit4]   2> 2449273 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54520_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2449284 INFO  (OverseerStateUpdate-97469122420670469-127.0.0.1:54514_-n_0000000000) [n:127.0.0.1:54514_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2449396 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2450307 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2450329 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2450397 WARN  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2450399 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2450420 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2450421 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-1-001/cores/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 2450421 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f152046
   [junit4]   2> 2450424 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=5.0546875, floorSegmentMB=2.1708984375, forceMergeDeletesPctAllowed=17.25462231611045, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2450443 WARN  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2450504 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2450504 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2450505 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2450505 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2450505 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6265900090466819]
   [junit4]   2> 2450506 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4f198e23[collection1] main]
   [junit4]   2> 2450508 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2450509 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2450509 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2450510 INFO  (searcherExecutor-7872-thread-1-processing-n:127.0.0.1:54520_ x:collection1 c:collection1) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4f198e23[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2450511 INFO  (coreLoadExecutor-7871-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559505967920447488
   [junit4]   2> 2450522 INFO  (coreZkRegister-7866-thread-1-processing-n:127.0.0.1:54520_ x:collection1 c:collection1) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2450522 INFO  (coreZkRegister-7866-thread-1-processing-n:127.0.0.1:54520_ x:collection1 c:collection1) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2450522 INFO  (coreZkRegister-7866-thread-1-processing-n:127.0.0.1:54520_ x:collection1 c:collection1) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:54520/collection1/
   [junit4]   2> 2450522 INFO  (coreZkRegister-7866-thread-1-processing-n:127.0.0.1:54520_ x:collection1 c:collection1) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2450522 INFO  (coreZkRegister-7866-thread-1-processing-n:127.0.0.1:54520_ x:collection1 c:collection1) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:54520/collection1/ has no replicas
   [junit4]   2> 2450529 INFO  (coreZkRegister-7866-thread-1-processing-n:127.0.0.1:54520_ x:collection1 c:collection1) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:54520/collection1/ shard1
   [junit4]   2> 2450645 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2450704 INFO  (coreZkRegister-7866-thread-1-processing-n:127.0.0.1:54520_ x:collection1 c:collection1) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2450820 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2451689 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-2-001/cores/collection1
   [junit4]   2> 2451690 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-2-001
   [junit4]   2> 2451692 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2451694 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2a5cde60{/,null,AVAILABLE}
   [junit4]   2> 2451695 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@262a709c{SSL,[ssl, http/1.1]}{127.0.0.1:54539}
   [junit4]   2> 2451696 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.Server Started @2456620ms
   [junit4]   2> 2451696 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=54539, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-2-001/cores}
   [junit4]   2> 2451696 ERROR (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2451697 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2451697 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2451697 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2451697 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-16T15:59:49.074Z
   [junit4]   2> 2451705 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2451705 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-2-001/solr.xml
   [junit4]   2> 2451732 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54482/solr
   [junit4]   2> 2451749 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54539_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2451754 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54539_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2451758 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54539_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54539_
   [junit4]   2> 2451765 INFO  (zkCallback-2464-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2451765 INFO  (zkCallback-2468-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2451765 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2451765 INFO  (zkCallback-2480-thread-2-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2451866 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54539_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-2-001/cores
   [junit4]   2> 2451866 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54539_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2451869 INFO  (OverseerStateUpdate-97469122420670469-127.0.0.1:54514_-n_0000000000) [n:127.0.0.1:54514_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2451984 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2451984 INFO  (zkCallback-2480-thread-2-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2452909 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2452920 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2453008 WARN  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2453010 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2453030 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2453030 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-2-001/cores/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-2-001/cores/collection1/data/]
   [junit4]   2> 2453030 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f152046
   [junit4]   2> 2453033 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=5.0546875, floorSegmentMB=2.1708984375, forceMergeDeletesPctAllowed=17.25462231611045, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2453049 WARN  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2453146 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2453146 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2453148 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2453148 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2453148 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6265900090466819]
   [junit4]   2> 2453149 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4d9f661b[collection1] main]
   [junit4]   2> 2453151 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2453151 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2453152 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2453153 INFO  (searcherExecutor-7883-thread-1-processing-n:127.0.0.1:54539_ x:collection1 c:collection1) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4d9f661b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2453153 INFO  (coreLoadExecutor-7882-thread-1-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559505970690785280
   [junit4]   2> 2453159 INFO  (coreZkRegister-7877-thread-1-processing-n:127.0.0.1:54539_ x:collection1 c:collection1) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 2453160 INFO  (updateExecutor-2477-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 2453160 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 2453160 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 2453161 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 2453161 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 2453161 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:54520/collection1/] and I am [https://127.0.0.1:54539/collection1/]
   [junit4]   2> 2453165 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:54520]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:54539_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 2453193 INFO  (qtp1867159047-15562) [n:127.0.0.1:54520_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 2453194 INFO  (qtp1867159047-15562) [n:127.0.0.1:54520_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 2453194 INFO  (qtp1867159047-15562) [n:127.0.0.1:54520_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:54539_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:54539","node_name":"127.0.0.1:54539_","state":"down"}
   [junit4]   2> 2453268 INFO  (zkCallback-2480-thread-2-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2453268 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2453970 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-3-001/cores/collection1
   [junit4]   2> 2453971 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-3-001
   [junit4]   2> 2453973 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2453976 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@184de2e1{/,null,AVAILABLE}
   [junit4]   2> 2453977 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2d821adb{SSL,[ssl, http/1.1]}{127.0.0.1:54549}
   [junit4]   2> 2453977 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.Server Started @2458901ms
   [junit4]   2> 2453977 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=54549, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-3-001/cores}
   [junit4]   2> 2453978 ERROR (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2453978 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2453978 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2453978 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2453978 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-16T15:59:51.355Z
   [junit4]   2> 2453982 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2453982 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-3-001/solr.xml
   [junit4]   2> 2454040 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54482/solr
   [junit4]   2> 2454057 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54549_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2454062 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54549_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2454067 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54549_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54549_
   [junit4]   2> 2454072 INFO  (zkCallback-2464-thread-1-processing-n:127.0.0.1:54514_) [n:127.0.0.1:54514_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2454072 INFO  (zkCallback-2468-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2454072 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2454072 INFO  (zkCallback-2480-thread-2-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2454073 INFO  (zkCallback-2487-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2454184 INFO  (zkCallback-2480-thread-2-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2454184 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2454186 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54549_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-3-001/cores
   [junit4]   2> 2454186 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [n:127.0.0.1:54549_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2454190 INFO  (OverseerStateUpdate-97469122420670469-127.0.0.1:54514_-n_0000000000) [n:127.0.0.1:54514_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2454198 INFO  (qtp1867159047-15562) [n:127.0.0.1:54520_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:54539_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:54539","node_name":"127.0.0.1:54539_","state":"recovering"}
   [junit4]   2> 2454198 INFO  (qtp1867159047-15562) [n:127.0.0.1:54520_    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 2454198 INFO  (qtp1867159047-15562) [n:127.0.0.1:54520_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:54539_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1005
   [junit4]   2> 2454295 INFO  (zkCallback-2480-thread-2-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2454295 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2454295 INFO  (zkCallback-2487-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2455208 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2455219 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2455299 WARN  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2455301 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2455321 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2455321 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-3-001/cores/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001/shard-3-001/cores/collection1/data/]
   [junit4]   2> 2455321 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f152046
   [junit4]   2> 2455324 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=5.0546875, floorSegmentMB=2.1708984375, forceMergeDeletesPctAllowed=17.25462231611045, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2455337 WARN  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2455408 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2455408 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2455409 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2455409 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2455410 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6265900090466819]
   [junit4]   2> 2455410 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@451e3440[collection1] main]
   [junit4]   2> 2455412 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2455413 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2455413 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2455414 INFO  (searcherExecutor-7894-thread-1-processing-n:127.0.0.1:54549_ x:collection1 c:collection1) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@451e3440[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2455415 INFO  (coreLoadExecutor-7893-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559505973062664192
   [junit4]   2> 2455421 INFO  (coreZkRegister-7888-thread-1-processing-n:127.0.0.1:54549_ x:collection1 c:collection1) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 2455421 INFO  (updateExecutor-2484-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 2455422 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 2455422 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 2455422 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 2455422 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 2455422 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:54520/collection1/] and I am [https://127.0.0.1:54549/collection1/]
   [junit4]   2> 2455427 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:54520]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:54549_&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 2455434 INFO  (qtp1867159047-15560) [n:127.0.0.1:54520_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 2455435 INFO  (qtp1867159047-15560) [n:127.0.0.1:54520_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 2455435 INFO  (qtp1867159047-15560) [n:127.0.0.1:54520_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:54549_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"https://127.0.0.1:54549","node_name":"127.0.0.1:54549_","state":"down"}
   [junit4]   2> 2455534 INFO  (zkCallback-2480-thread-2-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2455535 INFO  (zkCallback-2487-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2455534 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2455840 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2455840 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Wait for recoveries to finish - wait 30 for each attempt
   [junit4]   2> 2455840 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
   [junit4]   2> 2456443 INFO  (qtp1867159047-15560) [n:127.0.0.1:54520_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:54549_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"https://127.0.0.1:54549","node_name":"127.0.0.1:54549_","state":"recovering"}
   [junit4]   2> 2456443 INFO  (qtp1867159047-15560) [n:127.0.0.1:54520_    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 2456443 INFO  (qtp1867159047-15560) [n:127.0.0.1:54520_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:54549_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1009
   [junit4]   2> 2461199 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:54520/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 2461200 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=https://127.0.0.1:54539 START replicas=[https://127.0.0.1:54520/collection1/] nUpdates=1000
   [junit4]   2> 2461207 INFO  (qtp1867159047-15564) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 2461207 INFO  (qtp1867159047-15564) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2461208 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 2461208 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
   [junit4]   2> 2461208 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2461208 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2461208 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2461208 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 2461208 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 2461209 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 2461209 INFO  (recoveryExecutor-2478-thread-1-processing-n:127.0.0.1:54539_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 2461214 INFO  (zkCallback-2480-thread-2-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2461214 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2461214 INFO  (zkCallback-2487-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2463451 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:54520/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 2463451 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=https://127.0.0.1:54549 START replicas=[https://127.0.0.1:54520/collection1/] nUpdates=1000
   [junit4]   2> 2463457 INFO  (qtp1867159047-15560) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 2463457 INFO  (qtp1867159047-15560) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2463459 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 2463459 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
   [junit4]   2> 2463459 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2463459 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2463459 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2463459 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 2463460 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 2463460 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 2463460 INFO  (recoveryExecutor-2485-thread-1-processing-n:127.0.0.1:54549_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 2463465 INFO  (zkCallback-2480-thread-2-processing-n:127.0.0.1:54539_) [n:127.0.0.1:54539_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2463465 INFO  (zkCallback-2487-thread-1-processing-n:127.0.0.1:54549_) [n:127.0.0.1:54549_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2463465 INFO  (zkCallback-2474-thread-1-processing-n:127.0.0.1:54520_) [n:127.0.0.1:54520_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2463883 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
   [junit4]   2> 2463912 INFO  (qtp403623984-15522) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2463912 INFO  (qtp403623984-15522) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2463914 INFO  (qtp403623984-15522) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2463914 INFO  (qtp403623984-15522) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 2
   [junit4]   2> 2463930 INFO  (qtp1867159047-15563) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2463930 INFO  (qtp1867159047-15563) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2463933 INFO  (qtp1867159047-15563) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2463933 INFO  (qtp1867159047-15563) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:54520/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
   [junit4]   2> 2463969 INFO  (qtp807652609-15589) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2463970 INFO  (qtp807652609-15589) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2463969 INFO  (qtp619126351-15625) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2463970 INFO  (qtp619126351-15625) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2463970 INFO  (qtp807652609-15589) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2463970 INFO  (qtp807652609-15589) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:54520/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
   [junit4]   2> 2463971 INFO  (qtp619126351-15625) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2463971 INFO  (qtp619126351-15625) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:54520/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
   [junit4]   2> 2463972 INFO  (qtp1867159047-15557) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={_stateVer_=collection1:11&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 50
   [junit4]   2> 2463982 INFO  (qtp1867159047-15564) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=1
   [junit4]   2> 2463990 INFO  (qtp807652609-15587) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2463999 INFO  (qtp619126351-15623) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2466008 INFO  (qtp403623984-15523) [n:127.0.0.1:54514_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1559505984168132608)} 0 2
   [junit4]   2> 2466015 INFO  (qtp619126351-15624) [n:127.0.0.1:54549_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1559505984171278336&distrib.from=https://127.0.0.1:54520/collection1/&wt=javabin&version=2}{deleteByQuery=*:* (-1559505984171278336)} 0 1
   [junit4]   2> 2466015 INFO  (qtp807652609-15588) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1559505984171278336&distrib.from=https://127.0.0.1:54520/collection1/&wt=javabin&version=2}{deleteByQuery=*:* (-1559505984171278336)} 0 1
   [junit4]   2> 2466016 INFO  (qtp1867159047-15563) [n:127.0.0.1:54520_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1559505984171278336)} 0 6
   [junit4]   2> 2466027 INFO  (qtp807652609-15589) [n:127.0.0.1:54539_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54520/collection1/&wt=javabin&version=2}{add=[0 (1559505984187006976)]} 0 1
   

[...truncated too long message...]

7.0.0.1:54539}
   [junit4]   2> 2657395 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4b4ece1{/,null,UNAVAILABLE}
   [junit4]   2> 2657396 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.ChaosMonkey monkey: stop shard! 54549
   [junit4]   2> 2657397 INFO  (TEST-PeerSyncReplicationTest.test-seed#[95AAC0766C0D9AED]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54482 54482
   [junit4]   2> 2657400 INFO  (Thread-3114) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54482 54482
   [junit4]   2> 2661017 WARN  (Thread-3114) [    ] o.a.s.c.ZkTestServer Watch limit violations:
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2>
   [junit4]   2> 6 /solr/aliases.json
   [junit4]   2> 5 /solr/security.json
   [junit4]   2> 5 /solr/configs/conf1
   [junit4]   2>
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2>
   [junit4]   2> 6 /solr/clusterstate.json
   [junit4]   2> 6 /solr/clusterprops.json
   [junit4]   2> 4 /solr/collections/collection1/state.json
   [junit4]   2> 2 /solr/overseer_elect/election/97469122420670474-127.0.0.1:54520_-n_0000000001
   [junit4]   2> 2 /solr/collections/collection1/leader_elect/shard1/election/97469122420670474-core_node1-n_0000000000
   [junit4]   2>
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2>
   [junit4]   2> 209 /solr/overseer/collection-queue-work
   [junit4]   2> 45 /solr/overseer/queue
   [junit4]   2> 15 /solr/overseer/queue-work
   [junit4]   2> 6 /solr/collections
   [junit4]   2> 5 /solr/live_nodes
   [junit4]   2>
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=PeerSyncReplicationTest -Dtests.method=test -Dtests.seed=95AAC0766C0D9AED -Dtests.slow=true -Dtests.locale=de-CH -Dtests.timezone=Asia/Chongqing -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE  219s J1 | PeerSyncReplicationTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: timeout waiting to see all nodes active
   [junit4]    > at __randomizedtesting.SeedInfo.seed([95AAC0766C0D9AED:1DFEFFACC2F1F715]:0)
   [junit4]    > at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
   [junit4]    > at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
   [junit4]    > at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
   [junit4]    > at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2661021 INFO  (SUITE-PeerSyncReplicationTest-seed#[95AAC0766C0D9AED]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.PeerSyncReplicationTest_95AAC0766C0D9AED-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=de-CH, timezone=Asia/Chongqing
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=178297424,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [SolrMetricReporterTest, TestLMDirichletSimilarityFactory, TestAnalyzedSuggestions, TestZkChroot, MetricUtilsTest, CSVRequestHandlerTest, TestCloudPseudoReturnFields, SolrCloudExampleTest, TestCoreDiscovery, SolrCoreMetricManagerTest, TestCodecSupport, TestDynamicFieldCollectionResource, TestClassicSimilarityFactory, TestInPlaceUpdatesDistrib, TestExactStatsCache, RuleEngineTest, TestNonDefinedSimilarityFactory, TestSearchPerf, HdfsSyncSliceTest, BigEndianAscendingWordDeserializerTest, TestConfigsApi, TestXmlQParserPlugin, TestIndexingPerformance, TestRTimerTree, TestSolrXml, ClassificationUpdateProcessorFactoryTest, CacheHeaderTest, FieldAnalysisRequestHandlerTest, NoCacheHeaderTest, TermVectorComponentDistributedTest, FieldMutatingUpdateProcessorTest, TestSchemaSimilarityResource, VMParamsZkACLAndCredentialsProvidersTest, TestFieldCacheSanityChecker, TestMaxScoreQueryParser, TestUtils, TestFieldCacheWithThreads, BasicAuthIntegrationTest, TestScoreJoinQPScore, OverseerStatusTest, HdfsLockFactoryTest, TestPKIAuthenticationPlugin, InfixSuggestersTest, TestLocalFSCloudBackupRestore, HdfsWriteToMultipleCollectionsTest, TestWriterPerf, TestStressRecovery, SharedFSAutoReplicaFailoverTest, MultiThreadedOCPTest, UUIDUpdateProcessorFallbackTest, SolrJmxReporterTest, TestRawResponseWriter, SolrCoreCheckLockOnStartupTest, SortByFunctionTest, TestFiltering, CoreAdminHandlerTest, TestPHPSerializedResponseWriter, TestSolrCoreParser, HLLUtilTest, RAMDirectoryFactoryTest, BadIndexSchemaTest, TestCloudInspectUtil, TestHdfsBackupRestoreCore, OverseerModifyCollectionTest, TestInitParams, WrapperMergePolicyFactoryTest, TestMiniSolrCloudCluster, DistributedSuggestComponentTest, RemoteQueryErrorTest, LoggingHandlerTest, CoreMergeIndexesAdminHandlerTest, TestSolr4Spatial, TestConfigSetsAPIZkFailure, TestSolrQueryResponse, TestFuzzyAnalyzedSuggestions, TestManagedSchema, TestGeoJSONResponseWriter, SolrGangliaReporterTest, ChaosMonkeySafeLeaderTest, TestReversedWildcardFilterFactory, TestCustomDocTransformer, TestApiFramework, RequestHandlersTest, TestFieldCache, TestDeleteCollectionOnDownNodes, DataDrivenBlockJoinTest, TestSolrCoreProperties, ReturnFieldsTest, RollingRestartTest, LeaderElectionIntegrationTest, TestConfig, TestSchemaManager, TestDocSet, TestUseDocValuesAsStored, TestSmileRequest, UpdateLogTest, UpdateRequestProcessorFactoryTest, DefaultValueUpdateProcessorTest, TestBackupRepositoryFactory, TestClusterProperties, DocValuesMissingTest, TermVectorComponentTest, TestQueryUtils, CloudExitableDirectoryReaderTest, TestQueryTypes, TestLeaderElectionZkExpiry, TestImpersonationWithHadoopAuth, CloudMLTQParserTest, TestSearcherReuse, DistributedFacetPivotLongTailTest, TestBlobHandler, CursorMarkTest, HdfsDirectoryFactoryTest, CircularListTest, TestRecovery, TestStandardQParsers, FullSolrCloudDistribCmdsTest, URLClassifyProcessorTest, TestJavabinTupleStreamParser, TestSolrCoreSnapshots, OpenCloseCoreStressTest, TestBulkSchemaAPI, ClusterStateTest, TlogReplayBufferedWhileIndexingTest, TestNumericTerms32, TestIndexSearcher, MigrateRouteKeyTest, AutoCommitTest, DistribDocExpirationUpdateProcessorTest, TestManagedStopFilterFactory, SmileWriterTest, TestGraphTermsQParserPlugin, TestSubQueryTransformer, DeleteLastCustomShardedReplicaTest, ZkSolrClientTest, ResourceLoaderTest, ShufflingReplicaListTransformerTest, TestRealTimeGet, BitVectorTest, TestCopyFieldCollectionResource, HdfsChaosMonkeySafeLeaderTest, TestComplexPhraseQParserPlugin, UUIDFieldTest, DeleteReplicaTest, TestTrie, SolrInfoMBeanTest, TestComponentsName, SpellCheckCollatorTest, TestDistributedMissingSort, TestIntervalFaceting, TestShortCircuitedRequests, TestTestInjection, RankQueryTest, TestJettySolrRunner, HdfsRecoveryZkTest, UpdateParamsTest, GraphQueryTest, SpellPossibilityIteratorTest, TestConfigSetsAPI, TestSuggestSpellingConverter, HdfsTlogReplayBufferedWhileIndexingTest, DirectoryFactoryTest, StatsComponentTest, TestAddFieldRealTimeGet, AnalyticsMergeStrategyTest, HighlighterConfigTest, LeaderFailoverAfterPartitionTest, TestSolrCloudWithSecureImpersonation, TestPathTrie, CollectionsAPIAsyncDistributedZkTest, BadComponentTest, TestDistribIDF, BasicDistributedZkTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, SyncSliceTest, TestRandomFaceting, ZkCLITest, TestDistributedGrouping, TestFaceting, ZkControllerTest, TestReload, TestRangeQuery, TestCoreContainer, SpellCheckComponentTest, QueryElevationComponentTest, ConvertedLegacyTest, TestSort, BasicFunctionalityTest, DirectUpdateHandlerTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, SolrIndexSplitterTest, TestExtendedDismaxParser, TestFoldingMultitermQuery, SuggesterTSTTest, SuggesterTest, TestStressLucene, SuggesterWFSTTest, PolyFieldTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestAtomicUpdateErrorCases, DirectUpdateHandlerOptimizeTest, TestRemoteStreaming, DocValuesMultiTest, DistanceFunctionTest, IndexBasedSpellCheckerTest, StandardRequestHandlerTest, DirectSolrSpellCheckerTest, TestOmitPositions, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, IndexSchemaRuntimeFieldTest, IndexSchemaTest, MBeansHandlerTest, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, SOLR749Test, TestSolrIndexConfig, SolrIndexConfigTest, TestBinaryField, OutputWriterTest, NotRequiredUniqueKeyTest, EchoParamsTest, TestSweetSpotSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestFastWriter, TestJmxMonitoredMap, TestFastLRUCache, PreAnalyzedFieldTest, DateFieldTest, DOMUtilTest, TestDocumentBuilder, SliceStateTest, FileUtilsTest, CursorPagingTest, TestCrossCoreJoin, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, AliasIntegrationTest, CollectionReloadTest, ConcurrentDeleteAndCreateCollectionTest, ConnectionManagerTest, DeleteInactiveReplicaTest, DeleteStatusTest, DistributedQueueTest, DocValuesNotIndexedTest, ForceLeaderTest, LeaderInitiatedRecoveryOnShardRestartTest, MissingSegmentRecoveryTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, PeerSyncReplicationTest]
   [junit4] Completed [503/694 (1!)] on J1 in 219.30s, 1 test, 1 failure <<< FAILURES!

[...truncated 63252 lines...]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3838 - Still Unstable!

Policeman Jenkins Server-2
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3838/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter

Error Message:
Collection not found: withShardField

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: withShardField
        at __randomizedtesting.SeedInfo.seed([F53CABC0B3C6DE6F:A06C43521F3F119F]:0)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1376)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1072)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:232)
        at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11764 lines...]
   [junit4] Suite: org.apache.solr.cloud.CustomCollectionTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/init-core-data-001
   [junit4]   2> 1475155 INFO  (SUITE-CustomCollectionTest-seed#[F53CABC0B3C6DE6F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 1475160 INFO  (SUITE-CustomCollectionTest-seed#[F53CABC0B3C6DE6F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1475161 INFO  (SUITE-CustomCollectionTest-seed#[F53CABC0B3C6DE6F]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001
   [junit4]   2> 1475162 INFO  (SUITE-CustomCollectionTest-seed#[F53CABC0B3C6DE6F]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1475162 INFO  (Thread-2361) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1475162 INFO  (Thread-2361) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1475263 INFO  (SUITE-CustomCollectionTest-seed#[F53CABC0B3C6DE6F]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:59183
   [junit4]   2> 1475293 INFO  (jetty-launcher-2682-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1475293 INFO  (jetty-launcher-2682-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1475293 INFO  (jetty-launcher-2682-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1475293 INFO  (jetty-launcher-2682-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1475298 INFO  (jetty-launcher-2682-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5cc8365a{/solr,null,AVAILABLE}
   [junit4]   2> 1475300 INFO  (jetty-launcher-2682-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@665ca0f9{/solr,null,AVAILABLE}
   [junit4]   2> 1475301 INFO  (jetty-launcher-2682-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@40547320{/solr,null,AVAILABLE}
   [junit4]   2> 1475302 INFO  (jetty-launcher-2682-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1a7b56b4{/solr,null,AVAILABLE}
   [junit4]   2> 1475308 INFO  (jetty-launcher-2682-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3f55b6a4{HTTP/1.1,[http/1.1]}{127.0.0.1:59188}
   [junit4]   2> 1475308 INFO  (jetty-launcher-2682-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@20ffb419{HTTP/1.1,[http/1.1]}{127.0.0.1:59189}
   [junit4]   2> 1475308 INFO  (jetty-launcher-2682-thread-3) [    ] o.e.j.s.Server Started @1482294ms
   [junit4]   2> 1475308 INFO  (jetty-launcher-2682-thread-2) [    ] o.e.j.s.Server Started @1482294ms
   [junit4]   2> 1475309 INFO  (jetty-launcher-2682-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59188}
   [junit4]   2> 1475309 INFO  (jetty-launcher-2682-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d1affd0{HTTP/1.1,[http/1.1]}{127.0.0.1:59190}
   [junit4]   2> 1475309 INFO  (jetty-launcher-2682-thread-4) [    ] o.e.j.s.Server Started @1482295ms
   [junit4]   2> 1475309 INFO  (jetty-launcher-2682-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59190}
   [junit4]   2> 1475309 ERROR (jetty-launcher-2682-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1475309 INFO  (jetty-launcher-2682-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59189}
   [junit4]   2> 1475310 ERROR (jetty-launcher-2682-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1475310 INFO  (jetty-launcher-2682-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1475309 ERROR (jetty-launcher-2682-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1475310 INFO  (jetty-launcher-2682-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1475310 INFO  (jetty-launcher-2682-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1475310 INFO  (jetty-launcher-2682-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@740539e0{HTTP/1.1,[http/1.1]}{127.0.0.1:59191}
   [junit4]   2> 1475310 INFO  (jetty-launcher-2682-thread-1) [    ] o.e.j.s.Server Started @1482296ms
   [junit4]   2> 1475310 INFO  (jetty-launcher-2682-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-17T02:19:11.785Z
   [junit4]   2> 1475310 INFO  (jetty-launcher-2682-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59191}
   [junit4]   2> 1475310 ERROR (jetty-launcher-2682-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1475310 INFO  (jetty-launcher-2682-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1475310 INFO  (jetty-launcher-2682-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1475310 INFO  (jetty-launcher-2682-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1475311 INFO  (jetty-launcher-2682-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-17T02:19:11.786Z
   [junit4]   2> 1475312 INFO  (jetty-launcher-2682-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1475313 INFO  (jetty-launcher-2682-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1475313 INFO  (jetty-launcher-2682-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1475313 INFO  (jetty-launcher-2682-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-17T02:19:11.788Z
   [junit4]   2> 1475314 INFO  (jetty-launcher-2682-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1475314 INFO  (jetty-launcher-2682-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1475314 INFO  (jetty-launcher-2682-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1475314 INFO  (jetty-launcher-2682-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-17T02:19:11.789Z
   [junit4]   2> 1475329 INFO  (jetty-launcher-2682-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1475330 INFO  (jetty-launcher-2682-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1475340 INFO  (jetty-launcher-2682-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1475340 INFO  (jetty-launcher-2682-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1475389 INFO  (jetty-launcher-2682-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59183/solr
   [junit4]   2> 1475389 INFO  (jetty-launcher-2682-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59183/solr
   [junit4]   2> 1475393 INFO  (jetty-launcher-2682-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59183/solr
   [junit4]   2> 1475409 INFO  (jetty-launcher-2682-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59183/solr
   [junit4]   2> 1475457 INFO  (jetty-launcher-2682-thread-2) [n:127.0.0.1:59188_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1475458 INFO  (jetty-launcher-2682-thread-3) [n:127.0.0.1:59189_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1475458 INFO  (jetty-launcher-2682-thread-2) [n:127.0.0.1:59188_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59188_solr
   [junit4]   2> 1475459 INFO  (jetty-launcher-2682-thread-1) [n:127.0.0.1:59191_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1475461 INFO  (jetty-launcher-2682-thread-2) [n:127.0.0.1:59188_solr    ] o.a.s.c.Overseer Overseer (id=97471558558679048-127.0.0.1:59188_solr-n_0000000000) starting
   [junit4]   2> 1475463 INFO  (jetty-launcher-2682-thread-3) [n:127.0.0.1:59189_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59189_solr
   [junit4]   2> 1475468 INFO  (jetty-launcher-2682-thread-1) [n:127.0.0.1:59191_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59191_solr
   [junit4]   2> 1475469 INFO  (jetty-launcher-2682-thread-4) [n:127.0.0.1:59190_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1475471 INFO  (zkCallback-2700-thread-1-processing-n:127.0.0.1:59189_solr) [n:127.0.0.1:59189_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1475473 INFO  (zkCallback-2706-thread-1-processing-n:127.0.0.1:59190_solr) [n:127.0.0.1:59190_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1475473 INFO  (zkCallback-2701-thread-1-processing-n:127.0.0.1:59191_solr) [n:127.0.0.1:59191_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1475474 INFO  (zkCallback-2699-thread-1-processing-n:127.0.0.1:59188_solr) [n:127.0.0.1:59188_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1475486 INFO  (jetty-launcher-2682-thread-4) [n:127.0.0.1:59190_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59190_solr
   [junit4]   2> 1475487 INFO  (zkCallback-2701-thread-1-processing-n:127.0.0.1:59191_solr) [n:127.0.0.1:59191_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1475487 INFO  (zkCallback-2706-thread-1-processing-n:127.0.0.1:59190_solr) [n:127.0.0.1:59190_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1475487 INFO  (zkCallback-2700-thread-1-processing-n:127.0.0.1:59189_solr) [n:127.0.0.1:59189_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1475488 INFO  (zkCallback-2699-thread-1-processing-n:127.0.0.1:59188_solr) [n:127.0.0.1:59188_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1475507 INFO  (zkCallback-2699-thread-1-processing-n:127.0.0.1:59188_solr) [n:127.0.0.1:59188_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1475507 INFO  (zkCallback-2706-thread-1-processing-n:127.0.0.1:59190_solr) [n:127.0.0.1:59190_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1475507 INFO  (zkCallback-2700-thread-1-processing-n:127.0.0.1:59189_solr) [n:127.0.0.1:59189_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1475507 INFO  (zkCallback-2701-thread-1-processing-n:127.0.0.1:59191_solr) [n:127.0.0.1:59191_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1475518 INFO  (jetty-launcher-2682-thread-2) [n:127.0.0.1:59188_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59188_solr
   [junit4]   2> 1475522 INFO  (zkCallback-2700-thread-1-processing-n:127.0.0.1:59189_solr) [n:127.0.0.1:59189_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1475523 INFO  (zkCallback-2701-thread-1-processing-n:127.0.0.1:59191_solr) [n:127.0.0.1:59191_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1475523 INFO  (zkCallback-2706-thread-1-processing-n:127.0.0.1:59190_solr) [n:127.0.0.1:59190_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1475523 INFO  (zkCallback-2699-thread-1-processing-n:127.0.0.1:59188_solr) [n:127.0.0.1:59188_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1475647 INFO  (jetty-launcher-2682-thread-1) [n:127.0.0.1:59191_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node1/.
   [junit4]   2> 1475647 INFO  (jetty-launcher-2682-thread-4) [n:127.0.0.1:59190_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node4/.
   [junit4]   2> 1475651 INFO  (jetty-launcher-2682-thread-3) [n:127.0.0.1:59189_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node3/.
   [junit4]   2> 1475654 INFO  (jetty-launcher-2682-thread-2) [n:127.0.0.1:59188_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node2/.
   [junit4]   2> 1478047 INFO  (SUITE-CustomCollectionTest-seed#[F53CABC0B3C6DE6F]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1478050 INFO  (SUITE-CustomCollectionTest-seed#[F53CABC0B3C6DE6F]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59183/solr ready
   [junit4]   2> 1478093 INFO  (TEST-CustomCollectionTest.testRouteFieldForHashRouter-seed#[F53CABC0B3C6DE6F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRouteFieldForHashRouter
   [junit4]   2> 1478095 INFO  (TEST-CustomCollectionTest.testRouteFieldForHashRouter-seed#[F53CABC0B3C6DE6F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1478105 INFO  (qtp2009691747-12854) [n:127.0.0.1:59191_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=3&router.field=shard_s&name=routeFieldColl&action=CREATE&numShards=4&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1478110 INFO  (OverseerThreadFactory-5824-thread-1-processing-n:127.0.0.1:59188_solr) [n:127.0.0.1:59188_solr    ] o.a.s.c.CreateCollectionCmd Create collection routeFieldColl
   [junit4]   2> 1478236 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard1_replica1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1478237 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard2_replica2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1478257 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard4_replica2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard4&wt=javabin&version=2
   [junit4]   2> 1478262 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard4_replica1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard4&wt=javabin&version=2
   [junit4]   2> 1478264 INFO  (qtp968816651-12864) [n:127.0.0.1:59189_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard2_replica1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1478264 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard3_replica1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard3&wt=javabin&version=2
   [junit4]   2> 1478265 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard3_replica2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard3&wt=javabin&version=2
   [junit4]   2> 1478265 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard1_replica2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1478398 INFO  (zkCallback-2701-thread-1-processing-n:127.0.0.1:59191_solr) [n:127.0.0.1:59191_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1478398 INFO  (zkCallback-2706-thread-1-processing-n:127.0.0.1:59190_solr) [n:127.0.0.1:59190_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1478398 INFO  (zkCallback-2699-thread-3-processing-n:127.0.0.1:59188_solr) [n:127.0.0.1:59188_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1478398 INFO  (zkCallback-2700-thread-1-processing-n:127.0.0.1:59189_solr) [n:127.0.0.1:59189_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1479294 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1479294 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1479306 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard3_replica1] Schema name=cloud-dynamic
   [junit4]   2> 1479307 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard4_replica2] Schema name=cloud-dynamic
   [junit4]   2> 1479310 INFO  (qtp968816651-12864) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1479311 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1479314 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1479347 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1479358 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1479360 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard4_replica1] Schema name=cloud-dynamic
   [junit4]   2> 1479360 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard1_replica1] Schema name=cloud-dynamic
   [junit4]   2> 1479364 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard1_replica2] Schema name=cloud-dynamic
   [junit4]   2> 1479367 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1479370 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard3_replica2] Schema name=cloud-dynamic
   [junit4]   2> 1479411 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard2_replica2] Schema name=cloud-dynamic
   [junit4]   2> 1479434 INFO  (qtp968816651-12864) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard2_replica1] Schema name=cloud-dynamic
   [junit4]   2> 1479438 WARN  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard4_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1479439 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1479469 WARN  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard1_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1479469 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1479471 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard1_replica1' using configuration from collection routeFieldColl
   [junit4]   2> 1479472 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.c.SolrCore [[routeFieldColl_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node4/routeFieldColl_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node4/./routeFieldColl_shard1_replica1/data/]
   [junit4]   2> 1479472 WARN  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard4_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1479474 WARN  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard2_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1479475 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1479475 WARN  (qtp968816651-12864) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard2_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1479475 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard4_replica2' using configuration from collection routeFieldColl
   [junit4]   2> 1479476 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.SolrCore [[routeFieldColl_shard4_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node1/routeFieldColl_shard4_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node1/./routeFieldColl_shard4_replica2/data/]
   [junit4]   2> 1479476 INFO  (qtp968816651-12864) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1479477 INFO  (qtp968816651-12864) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard2_replica1' using configuration from collection routeFieldColl
   [junit4]   2> 1479477 WARN  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard1_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1479478 INFO  (qtp968816651-12864) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.c.SolrCore [[routeFieldColl_shard2_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node3/routeFieldColl_shard2_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node3/./routeFieldColl_shard2_replica1/data/]
   [junit4]   2> 1479478 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1479479 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard2_replica2' using configuration from collection routeFieldColl
   [junit4]   2> 1479479 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.c.SolrCore [[routeFieldColl_shard2_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node1/routeFieldColl_shard2_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node1/./routeFieldColl_shard2_replica2/data/]
   [junit4]   2> 1479480 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard1_replica2' using configuration from collection routeFieldColl
   [junit4]   2> 1479480 WARN  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard3_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1479480 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.c.SolrCore [[routeFieldColl_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node2/routeFieldColl_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node2/./routeFieldColl_shard1_replica2/data/]
   [junit4]   2> 1479481 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1479481 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1479482 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard3_replica2' using configuration from collection routeFieldColl
   [junit4]   2> 1479482 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard4_replica1' using configuration from collection routeFieldColl
   [junit4]   2> 1479482 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.SolrCore [[routeFieldColl_shard3_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node2/routeFieldColl_shard3_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node2/./routeFieldColl_shard3_replica2/data/]
   [junit4]   2> 1479482 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.c.SolrCore [[routeFieldColl_shard4_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node3/routeFieldColl_shard4_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node3/./routeFieldColl_shard4_replica1/data/]
   [junit4]   2> 1479487 WARN  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard3_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1479488 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1479489 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard3_replica1' using configuration from collection routeFieldColl
   [junit4]   2> 1479489 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.c.SolrCore [[routeFieldColl_shard3_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node4/routeFieldColl_shard3_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_F53CABC0B3C6DE6F-001/tempDir-001/node4/./routeFieldColl_shard3_replica1/data/]
   [junit4]   2> 1479647 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1479647 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1479651 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1479651 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1479652 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1479652 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1479661 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1479661 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1479662 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1479662 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1479663 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1479663 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1479665 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1479665 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1479666 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1479666 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1479669 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1479669 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1479673 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1479673 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1479674 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1479674 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1479677 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1479677 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1479680 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1479680 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1479685 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@3de6a4b5[routeFieldColl_shard2_replica2] main]
   [junit4]   2> 1479685 INFO  (qtp968816651-12864) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1479685 INFO  (qtp968816651-12864) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1479687 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1479687 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1479687 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1479688 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1479688 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1479688 INFO  (searcherExecutor-5836-thread-1-processing-n:127.0.0.1:59191_solr x:routeFieldColl_shard2_replica2 s:shard2 c:routeFieldColl) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.c.SolrCore [routeFieldColl_shard2_replica2] Registered new searcher Searcher@3de6a4b5[routeFieldColl_shard2_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1479689 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559544941682622464
   [junit4]   2> 1479692 INFO  (qtp968816651-12864) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1479692 INFO  (qtp968816651-12864) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1479693 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@290b16f3[routeFieldColl_shard3_replica2] main]
   [junit4]   2> 1479694 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1479695 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1479695 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1479695 INFO  (searcherExecutor-5838-thread-1-processing-n:127.0.0.1:59188_solr x:routeFieldColl_shard3_replica2 s:shard3 c:routeFieldColl) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.SolrCore [routeFieldColl_shard3_replica2] Registered new searcher Searcher@290b16f3[routeFieldColl_shard3_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1479695 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559544941688913920
   [junit4]   2> 1479698 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3a0caa9e[routeFieldColl_shard1_replica1] main]
   [junit4]   2> 1479699 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1479700 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1479700 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1479700 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559544941694156800
   [junit4]   2> 1479700 INFO  (searcherExecutor-5833-thread-1-processing-n:127.0.0.1:59190_solr x:routeFieldColl_shard1_replica1 s:shard1 c:routeFieldColl) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.c.SolrCore [routeFieldColl_shard1_replica1] Registered new searcher Searcher@3a0caa9e[routeFieldColl_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1479701 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@2e05fb3f[routeFieldColl_shard4_replica2] main]
   [junit4]   2> 1479701 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@af35903[routeFieldColl_shard3_replica1] main]
   [junit4]   2> 1479705 INFO  (qtp2019694437-12846) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard3: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1479705 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1479707 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1479707 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1479708 INFO  (searcherExecutor-5840-thread-1-processing-n:127.0.0.1:59190_solr x:routeFieldColl_shard3_replica1 s:shard3 c:routeFieldColl) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.c.SolrCore [routeFieldColl_shard3_replica1] Registered new searcher Searcher@af35903[routeFieldColl_shard3_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1479708 INFO  (qtp1391405782-12869) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559544941702545408
   [junit4]   2> 1479709 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1479711 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1479711 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1479711 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@20c397a0[routeFieldColl_shard1_replica2] main]
   [junit4]   2> 1479712 INFO  (qtp1391405782-12857) [n:127.0.0.1:59190_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1479713 INFO  (searcherExecutor-5834-thread-1-processing-n:127.0.0.1:59191_solr x:routeFieldColl_shard4_replica2 s:shard4 c:routeFieldColl) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.SolrCore [routeFieldColl_shard4_replica2] Registered new searcher Searcher@2e05fb3f[routeFieldColl_shard4_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1479713 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1479713 INFO  (qtp2009691747-12856) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559544941707788288
   [junit4]   2> 1479714 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1479714 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1479715 INFO  (qtp2019694437-12848) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559544941709885440
   [junit4]   2> 1479715 INFO  (searcherExecutor-5837-thread-1-processing-n:127.0.0.1:59188_solr x:routeFieldColl_shard1_replica2 s:shard1 c:routeFieldColl) [n:127.0.0.1:59188_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.c.SolrCore [routeFieldColl_shard1_replica2] Registered new searcher Searcher@20c397a0[routeFieldColl_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1479717 INFO  (qtp2009691747-12855) [n:127.0.0.1:59191_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1479717 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7e36d00a[routeFieldColl_shard4_replica1] main]
   [junit4]   2> 1479719 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1479720 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1479720 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1479720 INFO  (qtp968816651-12862) [n:127.0.0.1:59189_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559544941715128320
   [junit4]   2> 1479720 INFO  (searcherExecutor-5839-thread-1-processing-n:127.0.0.1:59189_solr x:routeFieldColl_shard4_replica1 s:shard4 c:routeFieldColl) [n:127.0.0.1:59189_solr c:routeFieldColl